These are chat archives for nextflow-io/nextflow

23rd
Feb 2015
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:02
does storeDir retain both copies of the data? (http://www.nextflow.io/docs/latest/process.html#storedir)
Paolo Di Tommaso
@pditommaso
Feb 23 2015 20:07
both? In what meaning?
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:08
it sounds as though from the example provided that the output generated by the process will exist in both the working directory as well as in storeDir=='/db/genomes'
Paolo Di Tommaso
@pditommaso
Feb 23 2015 20:11
ah I see
I have to say that I don't remember perfectly, but it's likely
because it is supposed to work as a persisted cache
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:13
hm
my pipeline is downloading a bunch of large files from S3
Id like to cache them so I dont need to run that step every time
(-resume is another option)
but im just wondering if some combination of storeDir and scratch might do the trick
Paolo Di Tommaso
@pditommaso
Feb 23 2015 20:15
I see you use a process with storeDir to download files from S3 so that that stage is skipped next time, rightt
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:16
well im thinking about it
right now im basically just using resume
Paolo Di Tommaso
@pditommaso
Feb 23 2015 20:17
let me check the code
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:20
k
Paolo Di Tommaso
@pditommaso
Feb 23 2015 20:23
yes, it copies the results from the work dir to the storeDir folder
maybe it can be improved in the future adding some options controlling how copy/move these files
in the meanwhile you could use afterScript to cleanup the workDir
Andrew Stewart
@andrewcstewart
Feb 23 2015 20:33
ah, true