These are chat archives for nextflow-io/nextflow

9th
Feb 2018
Vladimir Kiselev
@wikiselev
Feb 09 2018 09:27
Hi Paolo, on our LSF farm cluster we have a requirement when running bsub that selected[mem] = rusage[mem]. Is there any common practice on how to fix that for every process? I’ve tried defining perJobMemLimit parameter, which worked for some processes, however, some other processes still divide the memory (is it because of the cpu requirements?) and fail to start.
Vladimir Kiselev
@wikiselev
Feb 09 2018 09:58
So, I have this in my config:
executor {
    name = 'lsf'
    perJobMemLimit = 16.GB
}

process {
  queue = 'normal'
  clusterOptions = '-R "select[mem>16384]"'
}
but some of the processes start with this set of parameters:
#BSUB -q normal
#BSUB -W 02:00
#BSUB -M 8192
#BSUB -R "rusage[mem=8192]"
#BSUB -R "select[mem>16384]"
so that selected[mem] != rusage[mem]
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:00
perJobMemLimit is a boolean
hence
executor {
    name = 'lsf'
    perJobMemLimit = true
}
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:01
Oh...
Thanks a lot, will try now
sorry for my stupidity
oh no ! :)
this is not supposed to be stupidity
you should be able to just set memory option without the need for clusterOptions
ah no, you are right
I was missing the select stuff
maybe we should add that by default
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:15
Thanks! Hmm… I’ve changed config to this:
executor {
    name = 'lsf'
    memory = 16384
    perJobMemLimit = true
}

process {
  queue = 'normal'
  clusterOptions = '-R "select[mem>16384]"’
}
But still some processes start with this:
#BSUB -q normal
#BSUB -W 02:00
#BSUB -M 8192
#BSUB -R "rusage[mem=8192]"
#BSUB -R "select[mem>16384]”
and produce error in bsub
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:16
it should be
executor {  
    name = 'lsf'
    perJobMemLimit = true
}

process {
  queue = 'normal'
  memory = 16384
  clusterOptions = '-R "select[mem>16384]"'
}
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:18
ok, trying it now, thanks!
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:18
let me know
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:21
ok, it has changed, but now there is a different error: 0: MEMLIMT value should be a positive integer. Job not submitted.. Somehow memory was not read correctly:
#BSUB -q normal
#BSUB -W 02:00
#BSUB -M 0
#BSUB -R "rusage[mem=0]"
#BSUB -R "select[mem>16384]"
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:21
oh
using memory = 16384
NF interprets it as *bytes* !
use
memory = '16 GB'
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:30
cool, works now! Great many thanks!
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:30
good
now, if you want to be a pro you could do
process {
  queue = 'normal'
  memory = '16 GB'
  clusterOptions = { "-R \"select[mem>${task.memory.toMega()}]\"" }
}
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:31
haha, I just wanted to ask you about it! You are too smart! ;-)
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:31
:satisfied:
Vladimir Kiselev
@wikiselev
Feb 09 2018 10:35
amazing, works like a charm! Now I am ready to do some analysis! ;-) thanks a million Paolo
Paolo Di Tommaso
@pditommaso
Feb 09 2018 10:36
you are welcome
Vladimir Kiselev
@wikiselev
Feb 09 2018 11:54
Paolo, is there any recommendation on the minimum RAM required for NF?
Paolo Di Tommaso
@pditommaso
Feb 09 2018 11:55
1 GB could be the baseline for most applications
Vladimir Kiselev
@wikiselev
Feb 09 2018 11:55
Thanks!
Paolo Di Tommaso
@pditommaso
Feb 09 2018 11:55
then it may depends how big is your workflow
Vladimir Kiselev
@wikiselev
Feb 09 2018 11:55
ok
Bioninbo
@Bioninbo
Feb 09 2018 11:57
Hello. I want to normalize my samples by the control. I don't know how to get only the control. if (id == controlid) { set
Paolo Di Tommaso
@pditommaso
Feb 09 2018 11:57
@wikiselev you may want to profile it with Java Mission Control or a similar tool
Vladimir Kiselev
@wikiselev
Feb 09 2018 11:57
ok, thanks, will have a look
Bioninbo
@Bioninbo
Feb 09 2018 11:58
output: if (id == controlid) { file("*.bam") into bam_control }
Sorry for the confusing messages. I tried the code above but it do not work. What would be a simple way to do that?
Paolo Di Tommaso
@pditommaso
Feb 09 2018 11:59
no if in input/output file declaration is allowed
Bioninbo
@Bioninbo
Feb 09 2018 13:27
Ok thanks. I used when: instead.
Paolo Di Tommaso
@pditommaso
Feb 09 2018 13:29
:+1:
Vladimir Kiselev
@wikiselev
Feb 09 2018 13:53
Hi again, Paolo! One more memory question: we have a limit of just 512M for Java per user, so when I run NF it runs well until at some point it says something like:
Feb-09 13:51:24.088 [Actor Thread 28] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
Paolo Di Tommaso
@pditommaso
Feb 09 2018 13:54
512 MB ?!
Vladimir Kiselev
@wikiselev
Feb 09 2018 13:54
yep ;-)
Paolo Di Tommaso
@pditommaso
Feb 09 2018 13:54
I think you can't neither run whatsapp on your android phone with 512 MB :)
Vladimir Kiselev
@wikiselev
Feb 09 2018 13:55
hahaha
so, even though I have this in my .bashrc:
export NXF_OPTS="-Xmx512M -XX:+UseSerialGC”
it still fails
is there anything I can do except asking to increase the memory limit
Paolo Di Tommaso
@pditommaso
Feb 09 2018 13:56
I think no
Vladimir Kiselev
@wikiselev
Feb 09 2018 13:56
ok, I see, thanks!
Paolo Di Tommaso
@pditommaso
Feb 09 2018 13:56
wait
no sorry, I don't think there are much more options then increasing the mem
Vladimir Kiselev
@wikiselev
Feb 09 2018 13:58
Ok, thanks!