These are chat archives for nextflow-io/nextflow

6th
Aug 2018
tbugfinder
@tbugfinder
Aug 06 2018 07:20
@srynobio I've built custom AMI using packer. Did you see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/automated_image_cleanup.html : ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION
Brad Langhorst
@bwlang
Aug 06 2018 12:44

It seems that nextflow is not happy to be run within the same folder as another concurrent nextflow process… even with -w set to a unique location. will disabling cache help? or do i need mktmp for each execution?

I ended up making a directory tree continaing one directory for each nf workflow… anybody know of a simpler path?

Paolo Di Tommaso
@pditommaso
Aug 06 2018 13:04
this is the suggest approach however if you specify a different work-dir with the -w option you should not have any problem
Brad Langhorst
@bwlang
Aug 06 2018 13:12
hmm - even with -w set i see conflicts within .nextflow/… even if not i think .nextflow.log would end up being a confusion without the directory tree.
Paolo Di Tommaso
@pditommaso
Aug 06 2018 13:12
in that case you should specify a separate log file name with -log
Anthony Underwood
@aunderwo
Aug 06 2018 17:02
What's the current thinking about Nextflow and CWL?
I'm being asked to think about whether the pipelines we are developing should be platform agnostic
Do the limitations of CWL make the practicalities of running CWL in Nextflow challenging.
I know there is the https://github.com/nextflow-io/cwl2nxf project but wondered if this is likely to be supported
Paolo Di Tommaso
@pditommaso
Aug 06 2018 17:11
Too complex to fully support and partial support just useless
Shawn Rynearson
@srynobio
Aug 06 2018 18:02
@tbugfinder I've look into that and even-though the docs say the default is 3 hours, after reviewing the /etc/ecs/ecs.config file it set to 2m.
tbugfinder
@tbugfinder
Aug 06 2018 18:25
Is latest ECS agent in place
Shawn Rynearson
@srynobio
Aug 06 2018 18:30
yes. this is a weird case. I'm currently trying to debug if the issue is with the AWS container agent correctly cleaning up containers not running any tasks, or if it a communication issue between nf and aws-batch.
tbugfinder
@tbugfinder
Aug 06 2018 20:09
What's output of docker inspect ecs-agent|grep ECS ?
Brad Langhorst
@bwlang
Aug 06 2018 20:55

arrgh - i just looked at jobs that were supposed to run over the weekend they are sitting on the first step due to a missing dependency at the end of a pipe. pipefail would have caused those jobs to fail immediately and let me know. I’m adding

process.shell = ['/bin/bash','-o','pipefail']

as suggested by paolo above… but I wonder if this should be a default setting? Any thoughts on that?

Paolo Di Tommaso
@pditommaso
Aug 06 2018 21:16