These are chat archives for nextflow-io/nextflow

16th
Dec 2015
Michael L Heuer
@heuermh
Dec 16 2015 16:18
is there a way to spawn a daemon/service process, something that runs for the duration of the workflow and then gets shut down when the workflow completes?
Paolo Di Tommaso
@pditommaso
Dec 16 2015 17:06
Um, it depends what u need to do
At the very end any process is a background process
What daemon would you like to spawn?
Michael L Heuer
@heuermh
Dec 16 2015 18:06
A whole bunch of them. Hadoop HDFS master and worker nodes, Spark master and worker nodes, etc.
Michael L Heuer
@heuermh
Dec 16 2015 18:18
Specifically, I just need to docker run -d something when a process is executed, hold on to its container id, and then docker stop + docker rm when the workflow completes
Michael L Heuer
@heuermh
Dec 16 2015 21:09
We are using entry points, so this new feature could be useful http://www.nextflow.io/docs/latest/docker.html#executable-containers
Paolo Di Tommaso
@pditommaso
Dec 16 2015 21:16
@heuermh you could use onComplete event handler to shutdown the cluster
then, what's your idea? having nextflow processes to submit M/R or Spark tasks ?
Michael L Heuer
@heuermh
Dec 16 2015 21:33
hmm, if the processes spawning via docker run -d don't complete, then onComplete won't happen, right?
yep, I'd like to submit Spark jobs, and include setting up the Spark cluster as part of the workflow. Whether this actually works on top of the executors supported by Nextflow I'm not sure
Paolo Di Tommaso
@pditommaso
Dec 16 2015 21:57
onComplete is invoked when all processes have been executed
at the end it's almost the same of doing
launch-cluster.sh
nextflow run <pipeline>
shutdown-cluster.sh
however what you are trying to do looks very similar to the nextflow distributed execution I've blogged here http://www.nextflow.io/blog/2015/mpi-like-execution-with-nextflow.html