These are chat archives for nextflow-io/nextflow

26th
Jul 2018
KochTobi
@KochTobi
Jul 26 2018 08:31
Hi there, when I use aws and an ec2 cloud the correct AMi is now chosen. Everytime I use awsbatch the standard amazon linux image is used for the instances. I double checked my config files and cannot find the error. Any idea why it won't work with awsbatch? What did I do wrong?
Paolo Di Tommaso
@pditommaso
Jul 26 2018 08:34
when using awsbatch executor the AMI used depends by the compute environment associated to the batch queue
the cloud settings are ignored
KochTobi
@KochTobi
Jul 26 2018 08:36
Oh I see. Will try that out
Luca Cozzuto
@lucacozzuto
Jul 26 2018 10:11
hi Paolo! Where is the documentation about weblogs?
Luca Cozzuto
@lucacozzuto
Jul 26 2018 10:22
thanks!!!!
Thomas Van Parys
@thpar
Jul 26 2018 12:49
Hey guys. Would it be possible to set the publishDir mode by default in the config rather then specifying it for every process? Not every process in my pipeline does have a publishDir however.
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:50
yes, put in the nextflow.config file
process.publishDir = [path:'/some/path']
Thomas Van Parys
@thpar
Jul 26 2018 12:52
Hmm, no, not a default path, a default mode (link instead of the default symlink)
I think the example you gave would let all processes output into the same publish dir, no?
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:53
you are right, sorry I've misread
Thomas Van Parys
@thpar
Jul 26 2018 12:54
No problem
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:54
if so not possible right now, but it's a planned improvement
Thomas Van Parys
@thpar
Jul 26 2018 12:54
Ok, thanks. It's not a big deal to add it manually to any directive. Would've just been cleaner.
Paolo Di Tommaso
@pditommaso
Jul 26 2018 12:55
tend to agree
Alexander Peltzer
@apeltzer
Jul 26 2018 13:55
Anybody an idea when jobs hang in the ' Runnable ' Stage on AWSBatch?
Does that just mean no resources (yet) available?
Increased spot request to 100%, so that shouldn't really be an issue
Paolo Di Tommaso
@pditommaso
Jul 26 2018 13:56
I've seen job staying a long time in runnable status
Alexander Peltzer
@apeltzer
Jul 26 2018 13:57
Ok
I'll just wait - I have time :-)
Paolo Di Tommaso
@pditommaso
Jul 26 2018 13:57
check if a corresponding spot instance request exist
KochTobi
@KochTobi
Jul 26 2018 14:03
When I start a job, the job is stuck in runnable eventhough an ec2 instance was created by awsbatch and is running (finished initializing). @pditommaso can you confirm this observation with your jobs?
Alexander Peltzer
@apeltzer
Jul 26 2018 14:08
Hm, should at least keep the jobs as "Submitted" in the Dashboard... can't see spot request either
=========================================
[warm up] executor > awsbatch
Uploading local `bin` scripts folder to s3://XYZ/work/tmp/a9/aefb769b48adddf149678b0c895eff/bin
Log Says:
Jul-26 14:07:25.327 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > merge_featureCounts -- maxForks: 2
Jul-26 14:07:25.329 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Jul-26 14:07:25.329 [main] DEBUG nextflow.Session - Session await
(and then nothing happens)
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:12
are the jobs submitted in the batch dashboard ?
Alexander Peltzer
@apeltzer
Jul 26 2018 14:12
No, can't see them there either
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:12
uploaded the complete log
Alexander Peltzer
@apeltzer
Jul 26 2018 14:13
Ok
Vladimir Kiselev
@wikiselev
Jul 26 2018 14:34
Hi Paolo, is this still working?
k8s {
  cleanup = false
}
Looks like NF deletes all the failed pods...
Paolo Di Tommaso
@pditommaso
Jul 26 2018 14:40
if so open an issue please
Vladimir Kiselev
@wikiselev
Jul 26 2018 14:57
ok, thanks, will check again and open if needed