These are chat archives for nextflow-io/nextflow

13th
Jul 2017
Jean-Christophe Houde
@jchoude
Jul 13 2017 01:20
Hi all, it might be a very basic question, but I can't quite find an answer in the doc. To ease the use of a pipeline for some of our users, I want to provide a kind of "run-time" alias that would allow users to specify the maximum number of processes to launch without having to either change the config file (with the executor.local scope) or remembering the -executor.cpus syntax.
I see that I can change some settings on a process basis, in the directive section. However, I'm trying to parse the command-line argument at the beginning of the pipeline and affect that for the whole pipeline.
Trying something like:
if(params.processes > 0) {
$executor.$local.cpus = params.processes
}
doesn't work, and the other variations (not considering executor as a variable, etc) don't. I'm stuck, and just am wondering if my intent is even possible or if it isn't.
Paolo Di Tommaso
@pditommaso
Jul 13 2017 07:23
@jchoude executor settings can only be defined in the nextflow config file
something like the following should work
executor.$local.cpus = params.processes ?: Runtime.runtime.availableProcessors()
Robert Syme
@robsyme
Jul 13 2017 08:29
(Belated) thanks, Paolo!
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:30
welcome!
hope it helped
Robert Syme
@robsyme
Jul 13 2017 08:33
So great to see Pierre here!
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:34
I guess he won't last long ;)
Maxime Garcia
@MaxUlysse
Jul 13 2017 08:37
I see that I have missed interesting talks, but so sad to have missed Pierre here
Robert Syme
@robsyme
Jul 13 2017 08:42
If he doesn't come back, we'll know it was because he implemented NF as a XSLT transform.
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:43
ahahah
Pierre Lindenbaum
@lindenb
Jul 13 2017 08:48
:-)
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:50
all the web talk about your magic XSLT ability :)
Pierre Lindenbaum
@lindenb
Jul 13 2017 08:51
honestly, I'm struggling with the syntax. I know java 8 streams and each new NF statement is a mystery to me: is it NF specific ? is it groovy ? why does it fail ? I know I should read the full documentation... :-)
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:52
java 8 streams sounds similar but are not the same thing as NG channels
Maxime Garcia
@MaxUlysse
Jul 13 2017 08:52
@lindenb You're the only one to resist, as I read earlier, sounds like a familiar story
Happy to see you coming to this community ;-)
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:53
Have a look at this page
launch the nextflow console and play a bit with the snippets of code in that page
Pierre Lindenbaum
@lindenb
Jul 13 2017 08:54
I will
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:54
you will see is not that complex especially being familiar with java 8 streams
did you know JavaRx ?
Pierre Lindenbaum
@lindenb
Jul 13 2017 08:55
no
Paolo Di Tommaso
@pditommaso
Jul 13 2017 08:57
ok, just to mention it has been one of the first popular framework for reactive stream in java, most of the NF operators are inspired to that project
just forget if you don't know it
Jean-Christophe Houde
@jchoude
Jul 13 2017 10:10
@pditommaso Thanks it works!
Mike Smoot
@mes5k
Jul 13 2017 16:54
Hi @pditommaso I just had a pipeline stop while running with a slurm executor because sbatch failed to submit the file. I would have thought this would have been treated like a normal failure and retried (as configured in nextflow.config). Is sbatch outside of the normal failure tracking and retry mechanism?
Paolo Di Tommaso
@pditommaso
Jul 13 2017 17:11
interesting point
now, it is not but it make sense
please open an issue for that
Karin Lagesen
@karinlag
Jul 13 2017 18:02
is there any way from within nextflow to get a path to the profile file used, and all config files? I\d like to dump those to a publishDir for traceability purposes
Mike Smoot
@mes5k
Jul 13 2017 18:07
Thanks @pditommaso, I've opened a ticket: nextflow-io/nextflow#404
Paolo Di Tommaso
@pditommaso
Jul 13 2017 18:10
ok, thanks
@karinlag no, at this time it's only possible to get the profile name
Karin Lagesen
@karinlag
Jul 13 2017 18:15
ok, thanks!