These are chat archives for nextflow-io/nextflow

2nd
Jul 2016
Andrew Stewart
@astewart-twist
Jul 02 2016 21:11
@pditommaso Good memory! I have. What's very interesting about this time is that I now work at a completely different company!
so, different laptop, different network, different zipcode even :D
Anyone ever try running docker via nextflow via docker?
iow, nextflow running in a (privileged) container that runs workflows using the docker executor ?
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:13
:)
Andrew Stewart
@astewart-twist
Jul 02 2016 21:13
I can get nextflow to talk to the host docker daemon from within teh container no problem
but I think it's tripping up on the volume mounting part
would be cool if I can get it working :D
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:14
I wanted to try that, it should work
Andrew Stewart
@astewart-twist
Jul 02 2016 21:15
Im thinking that the work dir probably needs to be mounted from the host at the same absolute path
ie... /work:/work
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:15
Yes, that make sense
Andrew Stewart
@astewart-twist
Jul 02 2016 21:16
Ill try that real quick
(the other problem with this docker-in-docker method in general is matching the client API version with the server API version. makes portability a bit of an issue)
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:18
Indeed, I don't like it a lot
W
Andrew Stewart
@astewart-twist
Jul 02 2016 21:20
I think I came across a pure jvm implementation of a docker client
*docker api client
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:21
I think I saw it
However nxf just wrap the plain docker client
Andrew Stewart
@astewart-twist
Jul 02 2016 21:22
yeah, that's way easier
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:23
So, what's the point of your use case?
Andrew Stewart
@astewart-twist
Jul 02 2016 21:24
just being able to have nextflow orchestrate containerized workflow components from within docker itself
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:27
Using a resource manager such as SGE, etc?
Andrew Stewart
@astewart-twist
Jul 02 2016 21:27
im actually using celery / rmq for queueing
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:28
Ah, with nextflow?
Andrew Stewart
@astewart-twist
Jul 02 2016 21:32
yep
although the queueing mechanism isn't really important to the main idea
main idea is that nextflow makes a really nice analog to docker-compose
except for workflow compositions instead of service compositions
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:35
Um, than each container execution is scheduled through celery?
Andrew Stewart
@astewart-twist
Jul 02 2016 21:37
no just each nextflow execution
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:38
OK
Andrew Stewart
@astewart-twist
Jul 02 2016 21:38
so im using celery to schedule workflows, but each individual workflow is a composition of containerized processes
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:40
Anyway the trick is to map the docker socket in the container to the host docker socket
Andrew Stewart
@astewart-twist
Jul 02 2016 21:47
yup
or to expose the daemon api over https
BINGO
that worked
setting the nextflow workdir to the same absolute path on both host and container bind mount
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:52
:+1:
Andrew Stewart
@astewart-twist
Jul 02 2016 21:52
this is really awesome
and not terribly difficult to setup
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:53
Happy to know that it's working
Andrew Stewart
@astewart-twist
Jul 02 2016 21:54
I imagine this could work with SGE and other executors
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:55
Um, but then you should configure the SGE client in the nextflow container
Andrew Stewart
@astewart-twist
Jul 02 2016 21:55
yea
not that im trying to do that in this case, im just think of if this setup could work with the full range of nextflow's features
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:56
Makes sense
Andrew Stewart
@astewart-twist
Jul 02 2016 21:57
I think the next thing I want to try is running this on a docker-swarm cluster
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:58
I've tried that, it works like a charm
Andrew Stewart
@astewart-twist
Jul 02 2016 21:59
really
nice
Paolo Di Tommaso
@pditommaso
Jul 02 2016 21:59
But I've still not urder
Oops
Andrew Stewart
@astewart-twist
Jul 02 2016 22:00
I haven't done much with docker-swarm yet, but it looks like you could use it to distribute workflow processes similar to SGE
Paolo Di Tommaso
@pditommaso
Jul 02 2016 22:00
..still not understood if docker swarm manages the resources allocation
I mean CPUs, mem, etc
Have u any idea?
Andrew Stewart
@astewart-twist
Jul 02 2016 22:05
From what I understand, you can place resource constraints during container invocation, and in a swarm context the swarm daemon will schedule a container on a member node with the required resources
Paolo Di Tommaso
@pditommaso
Jul 02 2016 22:06
Ah that would be great