These are chat archives for nextflow-io/nextflow

26th
Jul 2018
KochTobi
@KochTobi
Jul 26 2018 08:31 UTC
Hi there, when I use aws and an ec2 cloud the correct AMi is now chosen. Everytime I use awsbatch the standard amazon linux image is used for the instances. I double checked my config files and cannot find the error. Any idea why it won't work with awsbatch? What did I do wrong?
Paolo Di Tommaso
@pditommaso
Jul 26 2018 08:34 UTC
when using awsbatch executor the AMI used depends by the compute environment associated to the batch queue
the cloud settings are ignored
KochTobi
@KochTobi
Jul 26 2018 08:36 UTC
Oh I see. Will try that out
Luca Cozzuto
@lucacozzuto
Jul 26 2018 10:11 UTC
hi Paolo! Where is the documentation about weblogs?
Luca Cozzuto
@lucacozzuto
Jul 26 2018 10:22 UTC
thanks!!!!
Thomas Van Parys
@thpar
Jul 26 2018 12:49 UTC
Hey guys. Would it be possible to set the publishDir mode by default in the config rather then specifying it for every process? Not every process in my pipeline does have a publishDir however.
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:50 UTC
yes, put in the nextflow.config file
process.publishDir = [path:'/some/path']
Thomas Van Parys
@thpar
Jul 26 2018 12:52 UTC
Hmm, no, not a default path, a default mode (link instead of the default symlink)
I think the example you gave would let all processes output into the same publish dir, no?
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:53 UTC
you are right, sorry I've misread
Thomas Van Parys
@thpar
Jul 26 2018 12:54 UTC
No problem
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:54 UTC
if so not possible right now, but it's a planned improvement
Thomas Van Parys
@thpar
Jul 26 2018 12:54 UTC
Ok, thanks. It's not a big deal to add it manually to any directive. Would've just been cleaner.
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:55 UTC
tend to agree
Alexander Peltzer
@apeltzer
Jul 26 2018 13:55 UTC
Anybody an idea when jobs hang in the ' Runnable ' Stage on AWSBatch?
Does that just mean no resources (yet) available?
Increased spot request to 100%, so that shouldn't really be an issue
Paolo Di Tommaso
@pditommaso
Jul 26 2018 13:56 UTC
I've seen job staying a long time in runnable status
Alexander Peltzer
@apeltzer
Jul 26 2018 13:57 UTC
Ok
I'll just wait - I have time :-)
Paolo Di Tommaso
@pditommaso
Jul 26 2018 13:57 UTC
check if a corresponding spot instance request exist
KochTobi
@KochTobi
Jul 26 2018 14:03 UTC
When I start a job, the job is stuck in runnable eventhough an ec2 instance was created by awsbatch and is running (finished initializing). @pditommaso can you confirm this observation with your jobs?
Alexander Peltzer
@apeltzer
Jul 26 2018 14:08 UTC
Hm, should at least keep the jobs as "Submitted" in the Dashboard... can't see spot request either
=========================================
[warm up] executor > awsbatch
Uploading local `bin` scripts folder to s3://XYZ/work/tmp/a9/aefb769b48adddf149678b0c895eff/bin
Log Says:
Jul-26 14:07:25.327 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > merge_featureCounts -- maxForks: 2
Jul-26 14:07:25.329 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Jul-26 14:07:25.329 [main] DEBUG nextflow.Session - Session await
(and then nothing happens)
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:12 UTC
are the jobs submitted in the batch dashboard ?
Alexander Peltzer
@apeltzer
Jul 26 2018 14:12 UTC
No, can't see them there either
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:12 UTC
uploaded the complete log
Alexander Peltzer
@apeltzer
Jul 26 2018 14:13 UTC
Ok
Vladimir Kiselev
@wikiselev
Jul 26 2018 14:34 UTC
Hi Paolo, is this still working?
k8s {
  cleanup = false
}
Looks like NF deletes all the failed pods...
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:40 UTC
if so open an issue please
Vladimir Kiselev
@wikiselev
Jul 26 2018 14:57 UTC
ok, thanks, will check again and open if needed