These are chat archives for nextflow-io/nextflow

31st
May 2016
Johan Viklund
@viklund
May 31 2016 09:12
I can't get the local executor to run in a vm with one core.
the trace file says "Scheduler queue size: 0"
if I allocate 2 cores to the vm it runs
is there anyway to get nextflow to run with only one cpu?
On the terminal the last thing I see is the [warm up] executor > local
Paolo Di Tommaso
@pditommaso
May 31 2016 09:18
oh, I've never seen this
what version of nextflow are you using?
Johan Viklund
@viklund
May 31 2016 09:19
hold on
Paolo Di Tommaso
@pditommaso
May 31 2016 09:19
nextflow info will tell you
Johan Viklund
@viklund
May 31 2016 09:20
0.19.2 build 3784
Paolo Di Tommaso
@pditommaso
May 31 2016 09:20
could you include the content of .nextflow.log ?
Johan Viklund
@viklund
May 31 2016 09:21
for the job that halts?
Paolo Di Tommaso
@pditommaso
May 31 2016 09:21
yes
Johan Viklund
@viklund
May 31 2016 09:21
ok, I just have to recreate the vm, should only take a few minutes
Paolo Di Tommaso
@pditommaso
May 31 2016 09:21
ok, no pb
there's the log
Paolo Di Tommaso
@pditommaso
May 31 2016 09:30
Ok, so basically it does not execute any job, right?
Johan Viklund
@viklund
May 31 2016 09:31
correct
Paolo Di Tommaso
@pditommaso
May 31 2016 09:31
do you have any process requiring more than one cpu?
Johan Viklund
@viklund
May 31 2016 09:32
this is the hello example from the website :)
Paolo Di Tommaso
@pditommaso
May 31 2016 09:32
:)
ok, I need to investigate that
is this blocking for you?
Johan Viklund
@viklund
May 31 2016 09:32
nope
I just configure Vagrant to setup 2 cores
I was a bit suprised though
Paolo Di Tommaso
@pditommaso
May 31 2016 09:33
me as well
I think it's a kind of regression due to a recent change in the local executor we did recently
thanks for reporting it
Johan Viklund
@viklund
May 31 2016 09:34
np
Johan Viklund
@viklund
May 31 2016 11:50
explicitly setting maxForks to 1 or higher does not resolve the issue (thought you might know)
Paolo Di Tommaso
@pditommaso
May 31 2016 11:52
I'm just uploading a minor release fixing the problem.
Indeed there was a bug
Johan Viklund
@viklund
May 31 2016 11:52
nice
now it will not be able to run more than one task at a time when using the local executor in case there is only one core?
Paolo Di Tommaso
@pditommaso
May 31 2016 11:55
exactly
Johan Viklund
@viklund
May 31 2016 11:55
I'm just thinking that when developing it might be good to be able to force it to run more things in parallel than the number of CPUs
Paolo Di Tommaso
@pditommaso
May 31 2016 11:56
the local executor it is also useful to deploy production pipeline to a single machine
Johan Viklund
@viklund
May 31 2016 11:57
yes
very much so
Paolo Di Tommaso
@pditommaso
May 31 2016 11:57
for this reason it is important that it apply the cpus and memory requirements correctly
Johan Viklund
@viklund
May 31 2016 11:58
Yes, I'm not sure how to do it to fit all the possible use cases
it's just, it would be nice if ;)
Paolo Di Tommaso
@pditommaso
May 31 2016 11:58
why you would like to launch more jobs than cpus in dev?
Johan Viklund
@viklund
May 31 2016 11:58
nextflow run --force-cpus=10 <pipeline>
it's mostly for doing some debugging I guess, just to make sure that I'm not making strange assumptions somewhere
haven't found the need for it yet though
consider it a might-be-nice-to-have feature request
but I understand that without use cases for it you will probably not spend time on it
Paolo Di Tommaso
@pditommaso
May 31 2016 12:00
in a traditional concurrency approach that could be useful to spot dead-locks or nondeterministic behaviour
however the dataflow parallelism used by nextflow is deterministic
Johan Viklund
@viklund
May 31 2016 12:01
hmm
yes that's true
Paolo Di Tommaso
@pditommaso
May 31 2016 12:03
in any case I agree there can be corner cases in which that can be useful, feel free to open an feature request on the nextflow GH repo
Johan Viklund
@viklund
May 31 2016 12:03
I'll do that once I have a real use case for it
Paolo Di Tommaso
@pditommaso
May 31 2016 12:03
ok
Paolo Di Tommaso
@pditommaso
May 31 2016 12:11
@viklund OK, the new release is available. Just update your installation using
nextflow -self-update
Johan Viklund
@viklund
May 31 2016 12:12
ah, thanks, that should work with the curl | sh one as well right?
Paolo Di Tommaso
@pditommaso
May 31 2016 12:12
sure
anyway is curl | bash
Johan Viklund
@viklund
May 31 2016 12:13
maybe you should put the curl url:s behind https to avoid man in the middle attacks
Paolo Di Tommaso
@pditommaso
May 31 2016 12:13
not sure that is working with sh
Johan Viklund
@viklund
May 31 2016 12:13
on centos sh==bash not on ubuntu though
Paolo Di Tommaso
@pditommaso
May 31 2016 12:13
makes sense, thanks for suggesting that