These are chat archives for nextflow-io/nextflow

12th
Jan 2017
Félix C. Morency
@fmorency
Jan 12 2017 14:41
@pditommaso I'm playing with Singularity and I was wondering how NF handles things when the image files are on a network share and the NF pipeline uses a scratch. Is there some sort of image caching on the local node to minimise impact on the network/NAS?
Paolo Di Tommaso
@pditommaso
Jan 12 2017 16:20
I guess it has been designed to work having the Singularity images on the shared file system
As far as I've understood the images doesn't need to be copied locally
Félix C. Morency
@fmorency
Jan 12 2017 16:26
Interesting.
Paolo Di Tommaso
@pditommaso
Jan 12 2017 16:27
Yes, it is designed having HPC in mind
Félix C. Morency
@fmorency
Jan 12 2017 18:07
After talking with the Singularity team, it seems that using a local image cache might or might not help depending on the image size and what % of the image is read. Also, files might stay in the page cache. The answer is very use-case specific.
Félix C. Morency
@fmorency
Jan 12 2017 20:42
@pditommaso from your experience, do you prefer having one (big) container with every dependencies in it or multiple (smaller) containers?