These are chat archives for nextflow-io/nextflow

7th
Sep 2018
Thomas Zichner
@zichner
Sep 07 2018 07:18
Hi Paolo and others,
According to #649 there is the possibility to clean up the work directory automatically upon successful pipeline completion. Is it somewhere documented how to use this feature?
Thank you very much
Paolo Di Tommaso
@pditommaso
Sep 07 2018 07:20
I guess no because it's somehow experimental, you should add
cleanup = true
in the config file
Thomas Zichner
@zichner
Sep 07 2018 07:21
Great! Thank you very much.
Paolo Di Tommaso
@pditommaso
Sep 07 2018 07:22
:+1:
Thomas Zichner
@zichner
Sep 07 2018 07:22
Can this also be combined with the keepLogs feature (#775)
Paolo Di Tommaso
@pditommaso
Sep 07 2018 07:23
no, this delete everything
Thomas Zichner
@zichner
Sep 07 2018 07:23
Ok. Good to know.
Still very useful!
Thanks a lot!
Paolo Di Tommaso
@pditommaso
Sep 07 2018 07:23
:+1:
micans
@micans
Sep 07 2018 09:51
I have this RUN wget -qO- https://get.nextflow.io | bash in a Dockerfile; can I combine this with getting a snapshot, e.g. 0.32.0u1-SNAPSHOT ?
Crabime
@Crabime
Sep 07 2018 09:55
how nextflow control memory and cpus. Here i met a scene, If i configure 2 cpu and 25G memory and running in Sun Grid Engine cluster, its occupy 50G memory totally. which i seen from sge. why not just 25G?
so as i see, nextflow occupying total memory=cpus * singleProcessMemory
Paolo Di Tommaso
@pditommaso
Sep 07 2018 10:30
No, NF uses the mem as you specified in the memory directory, does not take into account the num of cpus. See here.
@yes, define before export NXF_VER=0.32.0u1-SNAPSHOT
micans
@micans
Sep 07 2018 10:37
Cool :+1:
Maxime Garcia
@MaxUlysse
Sep 07 2018 10:39
@micans You're trying to run nextflow into docker? Will nextflow use other containers?
micans
@micans
Sep 07 2018 10:41
I'm making a nextflow container, running it on k8s, we will be starting pods from this container
Paolo Di Tommaso
@pditommaso
Sep 07 2018 10:41
@MaxUlysse there's nextflow -d run .. for that
micans
@micans
Sep 07 2018 10:45
@pditommaso was your last comment meant for me?
Oh I guess not
Paolo Di Tommaso
@pditommaso
Sep 07 2018 10:46
nope, I think Max was asking how to run docker in docker
micans
@micans
Sep 07 2018 10:46
we're creating a new container to have some more system stuff in it, e.g. perl. The nextflow kuberun login image is nice and tiny.
apologies for perl :-P
Paolo Di Tommaso
@pditommaso
Sep 07 2018 10:47
I see
Maxime Garcia
@MaxUlysse
Sep 07 2018 13:21
@pditommaso Oh yes, I remember you mentioned that non documented option
Crabime
@Crabime
Sep 07 2018 13:24
@pditommaso I want to know if nf will always output .exitcode file in its output directory
if server has no memory, will such file still be outputed?
Paolo Di Tommaso
@pditommaso
Sep 07 2018 13:26
.exitcode is always created unless the job is brutaly stopped ie. kill -9
Crabime
@Crabime
Sep 07 2018 13:28
great, so i can check whether job process properly just see .exitcode file content? is that right?
because a workmate tell me such file is not always right which made me really puzzled?
Paolo Di Tommaso
@pditommaso
Sep 07 2018 13:31
well, NF s doing that, not sure to understand why you need to check that file
Crabime
@Crabime
Sep 07 2018 13:33
I can check NF output and tell background program whether its process execute successfully or not
Crabime
@Crabime
Sep 07 2018 13:54
@pditommaso Hi Paolo, if i configure 2G memory to one process, while its memory exceed such value, what will NF do? if such process runs in SGE and exceed specified memory, will NF kill such process?
Paolo Di Tommaso
@pditommaso
Sep 07 2018 13:55
NF just set the job SGE directive, therefore it the cluster that kills it
Crabime
@Crabime
Sep 07 2018 13:56
If such process running in a local machine and exceed memory, will NF kill it?
Paolo Di Tommaso
@pditommaso
Sep 07 2018 13:56
no
Félix C. Morency
@fmorency
Sep 07 2018 14:18
Anyone here using NF on a local machine with some sort of "local scheduler/queue"? ie. launching multiple time NF using the local scheduler wont explode the computer resources / batch running multiple NF commands
I could probably use GNU parallel...
Shellfishgene
@Shellfishgene
Sep 07 2018 14:37
@fmorency Does the queueSize option not work for local?
Crabime
@Crabime
Sep 07 2018 15:56
@pditommaso Sorry bothering you again,I'm compiling NF source code and find there are some error in BigSort.java, FilesExMemoryUnit etc class can not find. but i can see nxf-common module already inside it through project structure view. and build process well using gradle. Do you know why?
as i run Launcher main function, it shows all classes inside common module can not find
I'm using IntelliJ Idea
Paolo Di Tommaso
@pditommaso
Sep 07 2018 16:20
have you imported the project properly ?
Anthony Underwood
@aunderwo
Sep 07 2018 17:17
@miepstei are you coming to the Workshop in Barcelona?