These are chat archives for nextflow-io/nextflow

8th
Feb 2017
Félix C. Morency
@fmorency
Feb 08 2017 14:55
@pditommaso How does NF + Ignite handles multiple users launching jobs on the head node?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:07
What exactly is your point?
Félix C. Morency
@fmorency
Feb 08 2017 15:13
Say I have 1 head node, 1 slave and 2 users that wants to launch jobs at the same time. Can we have multiple masters on the head node? Will it over-schedule jobs on the head node? Jobs will be processed in a FIFO manners?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:13
How are you launching the cluster? batch scheduler ?
Félix C. Morency
@fmorency
Feb 08 2017 15:13
Standard Ignite procedure in the NF documentation
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:14
NF cluster is meant to be transient ie. each pipeline is supposed to run its own cluster instance
Félix C. Morency
@fmorency
Feb 08 2017 15:17
Oh. You mean it's not meant to have an always-running slave waiting for jobs and pple just submitting stuff to run on the cluster?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:17
exactly
each pipeline should run its own set of daemon instances
Félix C. Morency
@fmorency
Feb 08 2017 15:19
Is there a way to achieve what I said ___^?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:21
usually batch schedulers provide a MPI interface, you can use that to launch the cluster instance
Félix C. Morency
@fmorency
Feb 08 2017 15:25
So is it safe to say that ^ is a layer on top of the cluster that manages the cluster resources and when a new user wants to schedule new jobs, that layer will check it resources are available and launch a new Ignite cluster instance with requested resources and execute the NF pipeline on that cluster instance. When pipeline is done, resources are freed. Is that right?
fmorency @fmorency is new to all this stuff
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:26
yes
this because NF is thought to be used on existing on-premises cluster that run SGE, SLURM, etc
or in the cloud
Félix C. Morency
@fmorency
Feb 08 2017 15:27
Ok. I'm building an on-premise cluster from scratch here. I'll read on how to deploy SLURM in that manner
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:27
in the latter you don't need to manage the resources, because you are supposed to launch a new bunch of VMs for each pipeline run
ok, take in consideration that NF can work directly with slurm w/o having to setup the Ignite daemon
Félix C. Morency
@fmorency
Feb 08 2017 15:30
Does it support the multi-user use case when used directly as I described above?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:30
yes, because jobs executions are delegated to SLURM
Félix C. Morency
@fmorency
Feb 08 2017 15:30
Ah great! Perfect
Thanks for all the info. Very useful.
Paolo Di Tommaso
@pditommaso
Feb 08 2017 15:31
:+1:
Félix C. Morency
@fmorency
Feb 08 2017 19:48
@pditommaso Did you test NF cloud + Singularity?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 19:49
nope
still thinking Docker is a better choice in the cloud.
Félix C. Morency
@fmorency
Feb 08 2017 19:55
In that case, Docker images are hosted on docker hub?
Paolo Di Tommaso
@pditommaso
Feb 08 2017 19:56
yes, you can even create an AMI with NF and Docker images preloaded to speed-up the cluster setup
but that's just an optimisation
Félix C. Morency
@fmorency
Feb 08 2017 19:57
Ok great