These are chat archives for nextflow-io/nextflow

14th
Sep 2018
Maxime Garcia
@MaxUlysse
Sep 14 2018 07:27
@fwip Travis is easier to set up, CircleCI has better machine
Tim Dudgeon
@tdudgeon
Sep 14 2018 11:55
How does one limit the number of tasks executing on an ignite node. I was thinking that -cluster.maxCpus 4 would do this, but it seems it doesn't limit the tasks, maybe just the cpus the tasks run on?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:02
-cluster.maxCpus 4 is doing that
micans
@micans
Sep 14 2018 12:14
afterScript 'sleep 1' fixes my Openstack glusterfs resume problem.
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:16
that's black magic, don't like it
micans
@micans
Sep 14 2018 12:17
so something to do with timestamps most likely?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:18
nearly finished a patch
micans
@micans
Sep 14 2018 12:19
I see an old bug in 2015 for glusterfs timestamps differing on different bricks. But what I see always seems completely consistent. Nothing is ever cached.
That 2015 bug is referenced later in a 2017 discussion still ongoing this year.
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:21
who is discussing about that ?
micans
@micans
Sep 14 2018 12:21
I'm looking at gluster/glusterfs#208
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:22
oh readhat people .. !
:joy:
micans
@micans
Sep 14 2018 12:22
I'm just browsing .. don't have a clue. What's your take?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:23
I'm adding an option to not include the timestamp when creating task hashes
micans
@micans
Sep 14 2018 12:24
Does that affect the utility much?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:24
I would say no
input file paths are supposed to be unique
micans
@micans
Sep 14 2018 12:25
perfect
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:25
process output files are unique by definition
Tim Dudgeon
@tdudgeon
Sep 14 2018 12:27
so starting the background process on a worker node like this nextflow node -bg -cluster.join path:/exports/nextflow/cluster -cluster.maxCpus 1 should make only one task execute at a time?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:29
yep
what does it mean ?
path:/exports/nextflow/cluster -cluster.maxCpus 1
Tim Dudgeon
@tdudgeon
Sep 14 2018 12:31
sorry, not clear what you are asking
Paolo Di Tommaso
@pditommaso
Sep 14 2018 12:42
not understanding this syntax path:/exports/nextflow/cluster -cluster.maxCpus 1
Félix C. Morency
@fmorency
Sep 14 2018 13:34
@micans cool! @pditommaso thanks for working on this :)
@pditommaso will the new lenient option resume correctly old runs that have been run using the workaround?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 13:46
will the new lenient option resume correctly old runs that have been run using the workaround?
not at all !
Tim Dudgeon
@tdudgeon
Sep 14 2018 13:51
@pditommaso The -cluster.join path:/exports/nextflow/cluster bit is how the node announces itself to the cluster (and is working fine) and the -cluster.maxCpus 1 bit is trying to limit to one task on the node. All as described here: https://www.nextflow.io/docs/latest/ignite.html
Paolo Di Tommaso
@pditommaso
Sep 14 2018 13:52
ah, yes, now I'm understanding
Félix C. Morency
@fmorency
Sep 14 2018 14:06
@pditommaso care to explain why?
Paolo Di Tommaso
@pditommaso
Sep 14 2018 14:43
because when using lenient the timestamp is not included therefore the hash is different