These are chat archives for nextflow-io/nextflow

28th
Apr 2015
Paolo Di Tommaso
@pditommaso
Apr 28 2015 06:18
What version of nextflow did you use in this try?
also are you using docker?
Michael L Heuer
@heuermh
Apr 28 2015 15:32
Version: 0.13.3 build 2909; this particular run was using docker, the one above was using the local executor.
Paolo Di Tommaso
@pditommaso
Apr 28 2015 16:07
Ok
Would it be possible for you to make a tarball of the pipeline workdir, the log file and eventually the dataset you use ?
Michael L Heuer
@heuermh
Apr 28 2015 19:37
Sure, we're trying to find a good test case that demonstrates the issue.
Paolo Di Tommaso
@pditommaso
Apr 28 2015 19:38
good
Michael L Heuer
@heuermh
Apr 28 2015 19:38
With slurm executor things appear to be working better. I'm still curious though, is there something magic about the number 32, in terms of jobs being submitted by the nextflow scheduler?
E.g. right now we have 32 PENDING jobs on our slurm queue
Paolo Di Tommaso
@pditommaso
Apr 28 2015 19:39
um, do you have a 32 cores machine/
?
Michael L Heuer
@heuermh
Apr 28 2015 19:43
Yep
Paolo Di Tommaso
@pditommaso
Apr 28 2015 19:45
thus, the local executor has processing queue of 32 jobs
actually 31
Michael L Heuer
@heuermh
Apr 28 2015 19:52
ah ok, that would explain it
Michael L Heuer
@heuermh
Apr 28 2015 20:51

Unrelated question, when using docker, if the process inside of docker runs as root, wouldn't it then be interacting with the nextflow working directories on the host filesystem as root?

To address this we create a nextflow user in the dockerfile with the same uid and groupid as a nextflow user on the host, but that doesn't seem like the right thing to do.

Paolo Di Tommaso
@pditommaso
Apr 28 2015 20:53
yes, I've noticed that and I didn't manage to use your container in that way
Usually I simply use the root user, the only side effect is that the files created by the container belongs to the root user
but that's not so bad because in principle you should never access in write mode
but do you run nextflow itself in the container ?
Michael L Heuer
@heuermh
Apr 28 2015 21:09
Yeah, that's the side effect we were trying to avoid. What do you mean by never access in write mode?
Paolo Di Tommaso
@pditommaso
Apr 28 2015 21:10
you can avoid that by using process.scratch=true
I mean that files written by a process are only read by other downstream processes
thus it's not an issue