These are chat archives for nextflow-io/nextflow

16th
Apr 2015
Andrew Stewart
@andrewcstewart
Apr 16 2015 00:27
Also I tried -w s3:// and my process seems to be hanging. As in, the processes run but nextflow never terminates.
Paolo Di Tommaso
@pditommaso
Apr 16 2015 07:31
You have found a bug, I'm going to release a fix for it.
while -w s3://something no. that works only when using Cirrus
Paolo Di Tommaso
@pditommaso
Apr 16 2015 17:18
@andrewcstewart Ok, it works with the latest version nextflow -self-update
Andrew Stewart
@andrewcstewart
Apr 16 2015 18:28
woo
will check out
huh.. I mean, -w s3:// did save the workdir files to S3
the pipeline just didnt terminate
Paolo Di Tommaso
@pditommaso
Apr 16 2015 18:35
yes, but -w s3:// it only work with cirrus executor, otherwise the pipeline hung
Andrew Stewart
@andrewcstewart
Apr 16 2015 18:37
Do you plan on expanding beyond cirrus?
Paolo Di Tommaso
@pditommaso
Apr 16 2015 18:37
um, not at this time
also because EFS is coming to Aws
Andrew Stewart
@andrewcstewart
Apr 16 2015 18:38
Just curious, but what causes the s3 workdir to depend on cirrus?
Paolo Di Tommaso
@pditommaso
Apr 16 2015 18:40
because when using s3 workdir inputs/outputs need to be uploaded to and downloaded from S3
and to do that I'm using a Cirrus tool
It could be possible in some other way, but it does not have a lot of sense outside the cloud
Andrew Stewart
@andrewcstewart
Apr 16 2015 18:42
ah
yet like I said, the workdir is indeed synching to s3
I guess my question is what is causing the stall?
(btw, s3 file input test worked)
Paolo Di Tommaso
@pditommaso
Apr 16 2015 18:43
that's cool, isn't it ? :)
it stalls because nextflow won't find the file signalling the end of the job
Andrew Stewart
@andrewcstewart
Apr 16 2015 19:02
think there's a way to spoof that?
or is there anything critical to the workdir/resume process that would be left out by a hack like that?
(I might have to dig up my Java IDE after all these long years of being a pythonista :smile: )
Paolo Di Tommaso
@pditommaso
Apr 16 2015 19:04
The main problem is this
you can't run a process on a remote dir
Andrew Stewart
@andrewcstewart
Apr 16 2015 19:05
(brb)
Andrew Stewart
@andrewcstewart
Apr 16 2015 19:29
what is the Cirrus tool you're referring to?
some kind of sync ?
Also, I notice that what's going on with the s3 input/ouputs is that there's a tmp directory created under /work... Do you think the location of that is potentially configurable?
because that could really help solve the situation of dealing with large data files from S3.. by tmp'ing them on say another mnt point
Paolo Di Tommaso
@pditommaso
Apr 16 2015 22:20
@andrewcstewart In cirrus platform is include a tool called es3 that allows you to copy/sync/etc files from/to s3 storage
Actually there are two temp folder used by nextflow. One of them is the one under the workdir. This cannot be configured, because it is required to be there.
The second is configure with the NXF_TEMP variable