These are chat archives for nextflow-io/nextflow

1st
Jun 2016
Paolo Di Tommaso
@pditommaso
Jun 01 2016 14:22
I don't want to give you bad news, but the lncRNA pipe does not work on my computer :/
This message was deleted
Evan Floden
@evanfloden
Jun 01 2016 14:23
with Docker?
Paolo Di Tommaso
@pditommaso
Jun 01 2016 14:24
This message was deleted
(wrong channel)
Paolo Di Tommaso
@pditommaso
Jun 01 2016 15:04
Hi @thejmazz, how is going the bionode streaming project ?
:)
Julian Mazzitelli
@thejmazz
Jun 01 2016 15:14
aha hi! :D
so far I am playing with make, snakemake and nextflow so I can see the strengths, weaknesses, unique features of each, and then we are going to define requirements on the JS prototype
Paolo Di Tommaso
@pditommaso
Jun 01 2016 15:17
sounds interesting
it would very useful if a some point you could blog/write your impressions about these tools
Julian Mazzitelli
@thejmazz
Jun 01 2016 15:19
yep that is my plan, its WIP. would you like to review it before I publish?
Paolo Di Tommaso
@pditommaso
Jun 01 2016 15:19
yes, sure
anyway, very proud you have chosen nextflow to take inspiration for your project
tx!
Mike Smoot
@mes5k
Jun 01 2016 18:02
Is there a way to access the executor queueSize within a nextflow script?
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:04
Um no, and it sounds a bad idea ;)
For what you need that?
Mike Smoot
@mes5k
Jun 01 2016 18:05
I was going to set the number of cpus as a function of the queueSize and available cpus on the machine.
We've got machines of very different sizes and I was thinking that it would be nice for a pipeline to autoscale to a given machine size.
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:10
If so I would suggest just to use Runtime.getRuntime(). availableProcessors() Java API
a bit verbose but does its work
does make sense?
Mike Smoot
@mes5k
Jun 01 2016 18:13
That's precisely what I was planning, but without the queueSize it won't do much for me. If I only have (say) 16 cpus and a queueSize of 8, then I'd like to be able to tell blast not to start more than 2 threads.
Ultimately I'd like to use as much of a machine's resources as possible without overwhelming it.
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:14
I see, but how are you changing the queueSize parameter?
Mike Smoot
@mes5k
Jun 01 2016 18:17
using the -queue-size command line option. The user of the pipeline would need to understand the relationship between queueSize and cpus, but I don't think that's asking too much.
My original idea was to let the users specify a value between 0 and 1 to reflect the percent of the machine to consume.
Maybe these problems become less of an issue when running on a cluster?
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:22
take in consideration when using the cluster the queueSize is 100 (or 200 don't remember well)
anyway you could do something like this
params.maxProcs = Runtime.getRuntime(). availableProcessors()  

executor {
  $local {
      queueSize = params.maxProcs 
  }
}

process {
  cpus = params.maxProcs 
}
I mean this is supposed to be the nextflow.config file
in this way you can control the num of cpus using pipeline cli option
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:27
and use that param to set the queueSize only for the local executor and the processes' cpus
when using an executor other than local that setting is ignored
Mike Smoot
@mes5k
Jun 01 2016 18:28
Yeah, that makes a lot of sense. Didn't realize I could write actual code in the nextflow.config!
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:31
the config file is a nextflow pipeline swiss army knife ;)
Mike Smoot
@mes5k
Jun 01 2016 18:32
Very good to know, thanks for the guidance!
Paolo Di Tommaso
@pditommaso
Jun 01 2016 18:33
welcome!