These are chat archives for nextflow-io/nextflow

31st
Jul 2018
Jemma Nelson
@fwip
Jul 31 2018 00:33
Can you just rename it when the process that needs it goes to use it? like file query_file name 'query.fa' from proteins from https://www.nextflow.io/docs/latest/process.html#input-of-files
Mike Smoot
@mes5k
Jul 31 2018 02:18
I actually need the name associated with S3 location - these are paired end reads that need to be merged and trimmed.
Paolo Di Tommaso
@pditommaso
Jul 31 2018 11:51
it depends on what's the final usage of that files
Francesco Strozzi
@fstrozzi
Jul 31 2018 11:53
@mes5k good, we are starting to explore Packer….for AWS Batch configuration, we used CloudFormation extensively, to setup all the roles and configurations. A part from the initial learning curve, then we have found it quite simple to work with.
Alexander Peltzer
@apeltzer
Jul 31 2018 14:01
Ok I'll have a look at Terraform
Just lost again around 4 hours digging in IAM roles
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:01
LOL
Alexander Peltzer
@apeltzer
Jul 31 2018 14:02
Seriously, having to configure tons of things that feel like a pyramid scheme.... you just solve one thing and get another 5 to configure
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:03
yes, it can be intimidating
Alexander Peltzer
@apeltzer
Jul 31 2018 14:04
Frustrating is more the way I feel right now ;-)
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:04
anyhow, you needs three IAM roles
the service role, the instance role, and the spot fleet role
in the service role I have just the AWSBatchServiceRole policy
Alexander Peltzer
@apeltzer
Jul 31 2018 14:06
Same here
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:06
in instance role: AmazonS3FullAccess and AmazonEC2ContainerServiceforEC2Role
then spot fleet role, just AmazonEC2SpotFleetRole
then it's important the role of the user running NF
to try add to it AmazonEC2FullAccess, AWSBatchFullAccess, AmazonS3FullAccess, AmazonEC2ContainerServiceFullAccess
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:15
@apeltzer Did you keep notes about everything?
Alexander Peltzer
@apeltzer
Jul 31 2018 14:15
I am ;-)
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:16
Smart guy B-)
Alexander Peltzer
@apeltzer
Jul 31 2018 14:16
I'll write a more detailed info about it from scratch as we're intending to use it much more
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:16
Alexander Peltzer
@apeltzer
Jul 31 2018 14:16
I know :-) Checked it out but no success :-(
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:17
Reference your issues as well
Alexander Peltzer
@apeltzer
Jul 31 2018 14:17
?
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:19
the problems you had, and what you tried to fix them
Alexander Peltzer
@apeltzer
Jul 31 2018 14:20
Yup - I do... its just taking so much time to get this up - much more than I expected
Thanks to your document I didnt get a single error of yours ;-)
Paolo I thinnk it works
Good news
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:23
LOL
Alexander Peltzer
@apeltzer
Jul 31 2018 14:23
Bad news: I think it works with 0.30.2 and not with 0.31.0
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:24
um, that's weird
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:25

Thanks to your document I didnt get a single error of yours ;-)

I'm glad I was of some help

Alexander Peltzer
@apeltzer
Jul 31 2018 14:26
You were!
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:29
I'm planning to do that again with Sarek soonish, so I'm looking forward to read your future document ;-)
Alexander Peltzer
@apeltzer
Jul 31 2018 14:29
I'll polish things and probably post similar to what you did and cross-link to your blogpost
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:29
Thanks :-)
Alexander Peltzer
@apeltzer
Jul 31 2018 14:29
Probably going a bit more in detail on these IAM roles/policies, since that was the problem here
Mike Smoot
@mes5k
Jul 31 2018 14:31
@apeltzer I saw the same 0.30.2 vs. 0.31.0 behavior the other day (the pipeline would hang and never submit jobs). I've begun the process of debugging, but I haven't had enough time to create a reproducible test case.
Paolo Di Tommaso
@pditommaso
Jul 31 2018 14:32
does this happen also with the basic rnaseq-nf pipeline ?
Mike Smoot
@mes5k
Jul 31 2018 14:34
Haven't tried. I created this pipeline as a test case, but that one actually worked for me. I've only had a few minutes to look into things so far, but will dig deeper later today.
Maxime Garcia
@MaxUlysse
Jul 31 2018 14:35
@apeltzer That would be good, I think that was what caused an issue later one when we realised with @ewels that a role was reopening instances everytime and spending all our credits
Alexander Peltzer
@apeltzer
Jul 31 2018 14:59
@pditommaso I will try this for that - running into the "Too Many jobs" issue which was fixed with 0.31.0 so I'd love to use 0.31.0 ;-)
@mes5k let me know what you find out - need to leave right now :-(
Paolo Di Tommaso
@pditommaso
Jul 31 2018 15:00
use NXF_VER=0.31.1-SNAPSHOT
hand open a issue on NF repo if it hangs
Francesco Strozzi
@fstrozzi
Jul 31 2018 16:38
guys for me AWS Batch works without problems with the new 0.31
@MaxUlysse what do you see ?
@apeltzer yes I know, IAM it’s fantastic but it’s daunting at first. Took me a while also to sort this out the first time. My advice is, if you can, start with the most open possible permissions and then when everything works, you can cut out the superflous and just stick with the least permissive IAM configuration that works in your specific case
so as Paolo suggested, just give S3FullAccess and BatchFullAccess to the roles and to your user and see if it works
Timur Shtatland
@tshtatland
Jul 31 2018 21:43
Hi, I am just starting to use nextflow. I noticed that that it is possible for a process to be successful if the last command in a shell pipe succeeded, even if one of the commands in a pipe failed (using nf version 0.31.0.4885). Is it possible to prevent this by adding automatically by default set -o pipefail to .command.sh files? This would be similar to set -u and set -e that are already there. Thank you!
Jemma Nelson
@fwip
Jul 31 2018 23:44
If you are interested in that for your process, you should just be able to add it to the script section.