These are chat archives for nextflow-io/nextflow

25th
Sep 2017
Ghost
@ghost~598345d2d73408ce4f6ff925
Sep 25 2017 02:41
Is it possible to disable the -resume feature? Or somehow minimize the amount of temporary files used throughout the workflow?
Also, what's the best way to delete temporary files after a process is complete?
Phil Ewels
@ewels
Sep 25 2017 04:30
rm -rf ./work :)
We use publishDir to copy out the files that we want to keep, then remove the work directory when the workflow completes successfully..
Paolo Di Tommaso
@pditommaso
Sep 25 2017 06:34
The temporary files are the ones created by the commands in your pipelines, NF add little
*adds
Simone Baffelli
@baffelli
Sep 25 2017 14:51
Good afternoon. Quick question: is there an "established " way to combine buffer and collectFile?
I want to collect several files from a buffer into a single file
Félix C. Morency
@fmorency
Sep 25 2017 15:20
Mmm I have a memory '16GB' directive in my process, but when I submitted my jobs to SLURM, it scheduled more processes than it should have on the node with 72Gb RAM
Paolo Di Tommaso
@pditommaso
Sep 25 2017 15:24
@baffelli I think buffer().flatten().collectFile() should work
@fmorency well, NF only pass the mem requirement on the job def, then it SLURM role to schedule correctly
Félix C. Morency
@fmorency
Sep 25 2017 15:26
Mmmm I wonder what's the issue
Paolo Di Tommaso
@pditommaso
Sep 25 2017 15:29
check the task .command.run header, and verify the SLURM requirement there
Félix C. Morency
@fmorency
Sep 25 2017 15:29
Yes it is there
Paolo Di Tommaso
@pditommaso
Sep 25 2017 15:29
is it correct ?
Félix C. Morency
@fmorency
Sep 25 2017 15:30
yes
Paolo Di Tommaso
@pditommaso
Sep 25 2017 15:31
hence you are in the wrong forum :D
Félix C. Morency
@fmorency
Sep 25 2017 15:31
:P
NOTE: Enforcement of memory limits currently relies upon the task/cgroup plugin or enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected). In both cases memory use is based upon the job's Resident Set Size (RSS). A task may exceed the memory limit until the next periodic accounting sample.
interesting
it doesn't seem to reserve anything