These are chat archives for nextflow-io/nextflow

1st
May 2015
Michael L Heuer
@heuermh
May 01 2015 17:52
when running on a large slurm cluster (say 4 nodes of 32 CPUs each) how does nextflow know how many processes to submit to slurm at one time? it doesn't look like nextflow is saturating the slurm cluster as well as it could
nextflow is running on the head node (32 CPUs) at the moment, but it would be nice if it wasn't
Paolo Di Tommaso
@pditommaso
May 01 2015 17:55
it submits as many jobs as it can up to 50 (by default)
you can increase/decrease by using the queueSize param in the nextflow.config file
Michael L Heuer
@heuermh
May 01 2015 17:57
ah sweet, thank you
Paolo Di Tommaso
@pditommaso
May 01 2015 18:00
BTW I found a couple of bugs in 0.13.x version that may cause the pipeline hung you reported
I would suggest to downgrade to nextflow 0.12.5 until a new version is available
Michael L Heuer
@heuermh
May 01 2015 18:06
thanks, yeah we haven't seen any problems since moving to slurm, or maybe any problems are masked by the complexity of the cluster environment :smile:
Paolo Di Tommaso
@pditommaso
May 01 2015 18:07
Yes, it's related to a race condition, so it arise when using the local executor