These are chat archives for nextflow-io/nextflow

3rd
Apr 2017
Paolo Di Tommaso
@pditommaso
Apr 03 2017 07:29
@rachanaj NF merges all config file you specify on the command line. They are reported in the .nextflow.log file, you can check there
if still not working please open an issue on GitHub including a test case to replicate the problem
Maarten van Gompel
@proycon
Apr 03 2017 10:23
How many cpus does the local executor use by default? (all of them?).. and how can I access this configuration value from my nextflow script (to pass it to a multithreaded tool)?
Paolo Di Tommaso
@pditommaso
Apr 03 2017 10:30
yes, all of them. See here.
task cpus are controlled by the cpus directive.
Maarten van Gompel
@proycon
Apr 03 2017 10:31
ah right! thanks!
Paolo Di Tommaso
@pditommaso
Apr 03 2017 10:32
welcome
Maarten van Gompel
@proycon
Apr 03 2017 10:34
So when I don't specify a cpus directive and run with the local executor ${task.cpus} will be the system maximum?
assuming only one process is running
Maarten van Gompel
@proycon
Apr 03 2017 10:51
hm, no..that doesn't entirely do what I want yet .. it gets set to 1, I need to explicitly use the cpus directive and set it to the maximum I assume
Paolo Di Tommaso
@pditommaso
Apr 03 2017 10:53
Nope task.cpus is 1 by default
Maarten van Gompel
@proycon
Apr 03 2017 10:53
So how would I access the configuration value from my script?
Paolo Di Tommaso
@pditommaso
Apr 03 2017 10:53
Instead the local executor uses all available CPUs
So by default it executes are any tasks as the number of avail CPUs
Not clear what u r trying to do
Maarten van Gompel
@proycon
Apr 03 2017 10:56
hmm, I'm a bit confused yes... I have a multithreaded process I want to run.. right now I manually pass a params.threads but I think that's inelegant and bypassing the actual system, I'm looking for the elegant solution: https://github.com/LanguageMachines/PICCL/blob/master/ticcl.nf#L154
I want to request the maximum number of cpus there (although I want the user to be able to override it if needed)
Paolo Di Tommaso
@pditommaso
Apr 03 2017 10:58
Ok, use task.cpus in the script
Need to go, will continue later.. sorry
Maarten van Gompel
@proycon
Apr 03 2017 11:01
no problem, thanks for the help :)
Phil Ewels
@ewels
Apr 03 2017 11:35
@proycon - I think you may be able to use executor.cpus?
eg. process.cpus = executor.cpus (then ${task.cpus} in the process script)
That should tell each process to request the number of cpus available to the executor, then the process script will request the number of cpus available to the process.
Not tried it myself though, this is just a guess
Maarten van Gompel
@proycon
Apr 03 2017 11:40
I'll give it a try!
nope, no such variable executor
ah wait, that was for the configuration, not the script itself
but, even if that works.. I assume it would then apply to all processes? I only want it for a couple that are multithreaded themselves
Phil Ewels
@ewels
Apr 03 2017 11:45
Then you can set the config for only those processes (process.$myprocess.cpus) instead of the global process.cpus
Maarten van Gompel
@proycon
Apr 03 2017 11:45
ah, that sounds good!
Phil Ewels
@ewels
Apr 03 2017 11:45
then all other processes will have 1 cpu (or whatever you set)
note that the queue will be bottlenecked for those processes that use all cpus, but I guess that's fine :)
Maarten van Gompel
@proycon
Apr 03 2017 11:46
yep, those run pretty sequentially
hmm.. that didn't seem to work yet
no errors though
Maarten van Gompel
@proycon
Apr 03 2017 11:52
LanguageMachines/PICCL@d0e2424
Phil Ewels
@ewels
Apr 03 2017 11:54
Yup, that's what I had in mind. If it doesn't work then you'll have to wait for feedback from the heavyweights sorry :/
@pditommaso..? ;)
Maarten van Gompel
@proycon
Apr 03 2017 11:55
thanks anyway :)
no rush
Phil Ewels
@ewels
Apr 03 2017 11:56
Oh, one last idea.. Maaaybe it could work if you wrap the exector statement in wiggly brackets? {executor.cpus}
Then it will evaluate at run time, by which time it could be set properly?
It's a long shot / guess though!
Maarten van Gompel
@proycon
Apr 03 2017 11:56
can't hurt to try :)
..nope
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:10
lost the thread, let me know if you need help
Phil Ewels
@ewels
Apr 03 2017 12:15
Yup - is it possible to use executor.cpus when assigning cpus to a process? eg. process.cpus = executor.cpus (then ${task.cpus} in the process script)
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:16
nope
ok, the problem is to get the avail cpus, right?
Phil Ewels
@ewels
Apr 03 2017 12:16
hah, ok. I think that was the conclusion that we were coming to after @proycon tried it and it didn't work ;)
Correct :+1:
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:17
it get been found with Runtime.runtime.availableProcessors()
for example
process foo {
  cpus Runtime.runtime.availableProcessors()
  script:
  """
  commad -t ${task.cpus}
  """
}
Phil Ewels
@ewels
Apr 03 2017 12:20
ok, nice! Not sure I would have ever got to that.. :runner:
Maarten van Gompel
@proycon
Apr 03 2017 12:20
ah, that sounds like a good solution indeed
Phil Ewels
@ewels
Apr 03 2017 12:21
And presumably you can set that in the config script with process.cpus = { Runtime.runtime.availableProcessors() } ?
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:21
Stack overflow IS the solution
:)
And presumably you can set that in the config script with
easier
process.cpus = Runtime.runtime.availableProcessors()
you need a closure i.e. { .. } only to defer the evaluation
Phil Ewels
@ewels
Apr 03 2017 12:23
yeah, I guess this isn't going to change during the course of a pipeline run
Maarten van Gompel
@proycon
Apr 03 2017 12:23
that's probably the most elegant solution then
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:23
but in any case the nextflow.config file is a proper groovy script, you can use any Java/Groovy expression
Maarten van Gompel
@proycon
Apr 03 2017 12:24
is params already available in the config scope?
I might want to add an override there then
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:25
the same as in the main script
params.cpus = .. etc
Maarten van Gompel
@proycon
Apr 03 2017 12:25
perfect
Paolo Di Tommaso
@pditommaso
Apr 03 2017 12:25
:+1: