These are chat archives for nextflow-io/nextflow

24th
May 2016
Evan Floden
@evanfloden
May 24 2016 20:56
This message was deleted
Jason Byars
@jbyars
May 24 2016 21:25
I'm back scratching my head with my publishDir + when: + --resume issues with intermediate jobs dying. I put together a gist explaining my confusion if you have a minute https://gist.github.com/jbyars/15a1527a095e392e8087642f079ba4a6
Paolo Di Tommaso
@pditommaso
May 24 2016 22:03
I see
when using storeDir the execution is skipped but the output is produced as expected .
ideally it should work with a S3 path
Jason Byars
@jbyars
May 24 2016 22:05
right, that's my dilemma
if I use both storeDir and publishDir, what happens?
Paolo Di Tommaso
@pditommaso
May 24 2016 22:06
what if you keep then locally and move the result when the pipeline complete as workaround ?
well it should work, it will store that files twice, locally and on S3
Jason Byars
@jbyars
May 24 2016 22:07
it would functionally work, but then we are back to the problem where I have to be good a guessing how large the shared volume needs to be.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:08
cloud is not haven :/
Jason Byars
@jbyars
May 24 2016 22:09
Nope, but when your in house cluster is going down in flames it looks pretty good.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:09
please report the problem of storeDir and S3, I can give a try next week
Jason Byars
@jbyars
May 24 2016 22:10
will do. I'm trying to remember now, is storeDir supposed to be a move or a copy or a symlink?
Paolo Di Tommaso
@pditommaso
May 24 2016 22:10
copy when you are using scratch
like in the case of S3
but I guess you want to move to save space
Jason Byars
@jbyars
May 24 2016 22:12
link is fine too. If I'm not really using any additional space I'm ok.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:13
symlinks cannot work across different file systems
Jason Byars
@jbyars
May 24 2016 22:13
I've been playing with publishDir for the past week so I'm trying to remember clearly. Right, copy when using scratch is fine. Symlink when not using scratch, and it's still not too bad.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:14
actually I was thinking to add a cleanup directive to automatically remove temp files
Jason Byars
@jbyars
May 24 2016 22:14
Then I just have to use one of the strategies we discussed in the past to clean up the `storeDir when I hit the end of the line
I think a cleanup directive will be needed in some form.
If storeDir and publishDir can coexist, I think I am ok for now. For some reason I had it in my head they are mutually exclusive.
Allocating an extra TB of EBS storage for an afternoon isn't going to kill me.
Andrew Stewart
@astewart-twist
May 24 2016 22:18
@jbyars is EFS an option for you?
Jason Byars
@jbyars
May 24 2016 22:19
technically, yes. I can redeploy in Oregon, but all my data is in N.Virigina right now.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:19
I've asked for the EFS beta one year ago they never replied
@astewart-twist did you have access to it?
Jason Byars
@jbyars
May 24 2016 22:22
Do you have an academic sales rep for your territory? Bugging them seemed to help me get access.
At $0.30/GB I'm not highly motivated to use it.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:23
ah, quite expensive
Andrew Stewart
@astewart-twist
May 24 2016 22:23
I haven’t used it yet
had access through a former employer but we never got around to trying it
Paolo Di Tommaso
@pditommaso
May 24 2016 22:23
$0.30/GB per month?
In theory if you're really good about cleaning up after yourself, it's not too bad.
Andrew Stewart
@astewart-twist
May 24 2016 22:26
thats how id use it
quickly expandable by very temporary scratch
Paolo Di Tommaso
@pditommaso
May 24 2016 22:27
well, it would be enough workflow.onComplete { file('work').deleteDir() }
and store all results on S3
Andrew Stewart
@astewart-twist
May 24 2016 22:28
yea
escape-pod whatever results you value to S3
then blow up the rest
Jason Byars
@jbyars
May 24 2016 22:31
What I would really like to see is EFS as one of the storage options for cfncluster. It would make a lot more sense. Then when you blow away the cluster, the storage automatically cleans up.
Paolo Di Tommaso
@pditommaso
May 24 2016 22:32
Need to go, bye.
Jason Byars
@jbyars
May 24 2016 22:33
thanks for the help!