These are chat archives for nextflow-io/nextflow

1st
Apr 2016
Matthieu Foll
@mfoll
Apr 01 2016 10:02
Hi Paolo, I have two questions/requests for the LSF scheduler:
For the memory: you divide the requested memory per the number of requested cpu
Because as you have in your comment "LSF specify per-process (per-core) memory limit (in MB)"
However this is not necessarly how all LSF clusters are configured (include ours). You can also have the case where the limit is per-job and not per-process (see https://www.ibm.com/support/knowledgecenter/#!/SSETD4_9.1.3/lsf_config_ref/lsf.conf.lsb_job_memlimit.5.dita)
Could you add an option like “per-job-memory-limit” that I can set to avoid dividing by the number of cpus in this case?
Matthieu Foll
@mfoll
Apr 01 2016 11:29
The second thing is about resource reservation: in LSF the -M mem option will kill the job if the job uses more than mem MB of memory, but it doesn’t reserve this amount of memory. To avoid overcommitment, one can add the option -R "rusage[mem=64000]" for example to reserve 64GB (see https://www.ibm.com/support/knowledgecenter/#!/SSETD4_9.1.2/lsf_admin/resource_res_about.dita)
It would be useful to also add this when using the memory directive and not only the -M mem
Paolo Di Tommaso
@pditommaso
Apr 01 2016 12:42
@mfoll I didn't know that. I think it's possible to add a flag in the nextflow config file to control as you are suggesting
regarding the second request should be possible
please open two separate feature requests for them
Matthieu Foll
@mfoll
Apr 01 2016 12:43
ok thanks I’ll do it