These are chat archives for nextflow-io/nextflow

13th
Sep 2018
Thomas Van Parys
@thpar
Sep 13 2018 13:23
How would I collect() a list of lists? The originating channel gives me a series of [x,y] and I'd like to gather them into a single list [ [x,y] , [x,y], [x,y], [x,y] ] . Currently I'm getting [x,y,x,y,x,y,x,y]
Paolo Di Tommaso
@pditommaso
Sep 13 2018 14:20
use toList() instead
Thomas Van Parys
@thpar
Sep 13 2018 14:27
@pditommaso : thanks! It seems like this difference in behaviour is exactly the difference between collect() and toList()
Anthony Underwood
@aunderwo
Sep 13 2018 15:22
Is there a reason why sometimes resume does not recognise that some jobs have already been completed?
Paolo Di Tommaso
@pditommaso
Sep 13 2018 15:24
Anthony Underwood
@aunderwo
Sep 13 2018 15:24
There have been occasions when all 50 jobs for a step have completed and the next step fails but then on resume nextflow recognises that 30 (for example) jobs have been cached but resubmits the rest
This is on openstack with ceph
maybe that file system suffers from the same issue
Paolo Di Tommaso
@pditommaso
Sep 13 2018 15:26
I'll try to implement a patch soon
Anthony Underwood
@aunderwo
Sep 13 2018 15:32
@pditommaso :+1:
Vladimir Kiselev
@wikiselev
Sep 13 2018 16:25
I think we have the same problem on GlusterFS on Openstack, @micans can confirm?
Paolo Di Tommaso
@pditommaso
Sep 13 2018 16:26
good, so we have a wide range of infras to test the patch ;)
Vladimir Kiselev
@wikiselev
Sep 13 2018 16:27
:+1:
micans
@micans
Sep 13 2018 16:34
I have -resume starting everything from scratch again, indeed. No toy example yet. I'll look at that deep cache option. Is it useful if I run this version NXF_VER=0.31.2-SNAPSHOT?
And yes, that's on Openstack with GlusterFS
Paolo Di Tommaso
@pditommaso
Sep 13 2018 16:35
nothing changed regarding caching in recent snapshots
Félix C. Morency
@fmorency
Sep 13 2018 17:15
@aunderwo @wikiselev @micans Have you tried the workaround provided in the issue?
Anthony Underwood
@aunderwo
Sep 13 2018 17:17
@fmorency afterScript 'sleep 1' in the config?
Félix C. Morency
@fmorency
Sep 13 2018 17:18
@aunderwo Yes
micans
@micans
Sep 13 2018 17:19
Will try!
Pbb tomorrow
Anthony Underwood
@aunderwo
Sep 13 2018 17:19
Will give it a go when it happens next - most likely later!
Jemma Nelson
@fwip
Sep 13 2018 20:22
Does anyone have experience setting up a CI server for a nextflow pipeline, or recommendations on services to use? basically I want to nextflow run -profile test,docker on pull requests, to make sure they aren't breaking anything.
Paolo Di Tommaso
@pditommaso
Sep 13 2018 21:08
Sure thing, basic examples
for more advanced config you may want to have a look at pipelines developed by http://nf-co.re/ folks
Jemma Nelson
@fwip
Sep 13 2018 21:24
thanks!
Paolo Di Tommaso
@pditommaso
Sep 13 2018 21:24
:ok_hand: