These are chat archives for nextflow-io/nextflow

17th
Apr 2017
Karin Lagesen
@karinlag
Apr 17 2017 08:24
...is there any way of loading modules and setting paths for a group of processes?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 09:30
what do you mean ?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 09:40
@timdiels Yes, exec: runs in its own directory, you need to manage it properly in your code by using the task.workDir variable
Tim Diels
@timdiels
Apr 17 2017 11:01
Is there a helper for running shell in an exec or is it easiest to just use String.execute?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 11:02
why using exec: if you want to run an external command ?
you can use script: for that
Karin Lagesen
@karinlag
Apr 17 2017 11:18
@pditommaso I have three processes that all need the same modules and a specific path set for a program
I would prefer to be able to set that in the config file just once, to avoid duplication errors
on a different matter: i am now testing out my workflow on slurm.
when I run it with a local executor, everything runs fine
Paolo Di Tommaso
@pditommaso
Apr 17 2017 11:21

I would prefer to be able to set that in the config file just once

no, it's not possible to specific the config for a group of processes

Karin Lagesen
@karinlag
Apr 17 2017 11:22
however, when I try it out with slurm, I get past the first process (collate_data) for all the data sets, but it doesn\t start more than one copy of each of the three downstream processes
Paolo Di Tommaso
@pditommaso
Apr 17 2017 11:52
how many times is execute the process collate_data ?
Karin Lagesen
@karinlag
Apr 17 2017 11:55
I have five data sets, and it is running once for each of them
and that works
Paolo Di Tommaso
@pditommaso
Apr 17 2017 11:56
both on local and slurm ?
Karin Lagesen
@karinlag
Apr 17 2017 11:56
yes
Paolo Di Tommaso
@pditommaso
Apr 17 2017 11:57
can you share the produced stdout both for local and slurm execution ?
Karin Lagesen
@karinlag
Apr 17 2017 12:02
...which stdout would that be? aka where do I find it?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:02
I mean just what's printed on script when you run your pipeline
Karin Lagesen
@karinlag
Apr 17 2017 12:04
ah
will do in a sec, the slurm one is just finishing up
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:05
no pb
Karin Lagesen
@karinlag
Apr 17 2017 12:19
(ariba) karinlag@eris[Bifrost] nextflow -c conf/input_and_software_params.config run -resume spec_gene_finding.nf -profile standard
N E X T F L O W  ~  version 0.24.2
Launching `spec_gene_finding.nf` [nauseous_meninsky] - revision: 9375b82399
=================================================
 Specific gene finding with Ariba v0.20170415
=================================================
Reads                   : ../testdata/risk/*L00{1,2}_R{1,2}_001.fastq.gz
#files in read set      : 4
MLST Scheme used        : Escherichia coli#1
AMR database            : card
Virulence db            : vfdb_core
Results can be found in : results
=================================================

[warm up] executor > local
[f6/00e8e6] Submitted process > run_ariba_amr_prep
[00/74f25c] Submitted process > collate_data (2)
[7c/223e75] Submitted process > collate_data (1)
[cf/f3fd36] Submitted process > run_ariba_mlst_prep
[98/560992] Submitted process > collate_data (3)
[ab/7082e4] Submitted process > run_ariba_vir_prep
[cb/027539] Submitted process > collate_data (4)
[5a/ca5a38] Submitted process > collate_data (5)
[3b/67908f] Submitted process > run_ariba_amr_pred (1)
[e2/1e1824] Submitted process > run_ariba_amr_pred (2)
[ea/cc2fd7] Submitted process > run_ariba_mlst_pred (2)
[19/3d03df] Submitted process > run_ariba_mlst_pred (1)
[33/c18c67] Submitted process > run_ariba_amr_pred (3)
[6d/6bce47] Submitted process > run_ariba_amr_pred (4)
[b6/aa3912] Submitted process > run_ariba_amr_pred (2)
[56/3217df] Submitted process > run_ariba_amr_pred (1)
[88/410763] Submitted process > run_ariba_mlst_pred (3)
[94/69a7b7] Submitted process > run_ariba_mlst_pred (4)
[24/737a25] Submitted process > run_ariba_amr_pred (5)
[a9/2a6489] Submitted process > run_ariba_amr_pred (3)
[f6/4c0e44] Submitted process > run_ariba_amr_pred (4)
[d5/745c3e] Submitted process > run_ariba_mlst_pred (5)
[91/434061] Submitted process > run_ariba_amr_pred (5)
[c8/b7a0a3] Submitted process > run_ariba_amr_summarize
[91/1b2bf9] Submitted process > run_ariba_mlst_summarize
[d2/962dcf] Submitted process > run_ariba_vir_summarize
(ariba) karinlag@eris[Bifrost]
and for the slurm one:
(ariba) [karinlag@abel 20170417_ariba_test]$ nextflow -c input_and_software_params.config run /work/projects/nn9305k/software/Bifrost/spec_gene_finding.nf -profile nn9305k
N E X T F L O W  ~  version 0.23.4
Launching `/work/projects/nn9305k/software/Bifrost/spec_gene_finding.nf` [serene_pike] - revision: 9375b82399
=================================================
 Specific gene finding with Ariba v0.20170415
=================================================
Reads                   : /work/projects/nn9305k/ecoli/analysis/tmp/20170417_ariba_test/raw_data_link/*L00{1,2}_R{1,2}_001.fastq.gz
#files in read set      : 4
MLST Scheme used        : Escherichia coli#1
AMR database            : card
Virulence db            : vfdb_core
Results can be found in : results
=================================================

[warm up] executor > slurm
[e9/ab8eaf] Submitted process > run_ariba_amr_prep (1)
[76/71549f] Submitted process > run_ariba_mlst_prep (1)
[6f/69f661] Submitted process > run_ariba_vir_prep (1)
[d4/24ca77] Submitted process > collate_data (1)
[b7/1a1b0c] Submitted process > collate_data (2)
[f6/c05f4a] Submitted process > collate_data (3)
[1c/692c79] Submitted process > collate_data (4)
[0b/1ae12e] Submitted process > collate_data (5)
[eb/8101f4] Submitted process > run_ariba_mlst_pred (1)
[03/bc4d3c] Submitted process > run_ariba_amr_pred (1)
[16/2080dd] Submitted process > run_ariba_amr_pred (1)
the slurm one isn't quite done yet, but it hasn't spawned more slurm jobs, so
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:21
do you mean it's still running
Karin Lagesen
@karinlag
Apr 17 2017 12:21
it fires up multiple slurm jobs util the last amr_pred job, and then it just rounds them off, and no other pred jobs are started
yes
but it will finish in a moment
correction: it has finished now
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:21
no other jobs submitted ?
Karin Lagesen
@karinlag
Apr 17 2017 12:22
nope
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:22
ahh .. found the problem
N E X T F L O W ~ version 0.23.4
Karin Lagesen
@karinlag
Apr 17 2017 12:22
doh
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:22
you need to update NF in the cluster
Karin Lagesen
@karinlag
Apr 17 2017 12:22
ah
what was the command for doing that again?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:23
nextflow -self-update
Karin Lagesen
@karinlag
Apr 17 2017 12:23
thanks!
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:23
welcome!
Karin Lagesen
@karinlag
Apr 17 2017 12:24
thanks!
Paolo Di Tommaso
@pditommaso
Apr 17 2017 12:25
welcome!
:D
Karin Lagesen
@karinlag
Apr 17 2017 12:48
works, btw
Tim Diels
@timdiels
Apr 17 2017 13:30
@pditommaso I'm using exec because I need some coordination between process instances. I have n fasta files, each needs to be transformed to a decypher database, each fasta needs to be blasted against all databases, finally all databases should be deleted. So, I have a blast process which takes a fasta and a databaseFasta; first it creates the database if it's missing, then it blasts against it. When checking and creating the database, no other process should look for or create the same database. I could do this with a bunch of lock files, but synchronized + exec: seemed easier
Also, when creating the database, I first delete it in case its source fasta changed
Tim Diels
@timdiels
Apr 17 2017 13:36
But I'm beginning to see that exec loses many of the advantages that script has, such as the module directive
Tim Diels
@timdiels
Apr 17 2017 13:43
My initial design was
--fastas--> createDb --dbs--+> blast --dbs.untilComplete--> rmDb
        --------------------+        --hits---------------------->
but when you change blast and -resume a finished run, the databases will have been deleted and will not be recreated.
So I merged createDb into blast
Tim Diels
@timdiels
Apr 17 2017 13:51
Caching question: when process1 outputs only a string to process2, when process1 is rerun but returns the same string, will process2 rerun?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 15:21
But I'm beginning to see that exec loses many of the advantages that script
exec is designed to run java/groovy code only is a distributed manner
script runs an external command but it can run also some java/groovy code mainly for purpose to parametrise the command script
Caching question: when process1 outputs only a string to process2, when process1 is rerun but returns the same string, will process2 rerun?
no, as long as you re-run it with the -resume option
Paolo Di Tommaso
@pditommaso
Apr 17 2017 15:29
@karinlag good!
Tim Diels
@timdiels
Apr 17 2017 16:05
@pditommaso Thanks! Is it possible to add some cleanup code to nextflow clean?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 16:06
umm, a custom strategy ?
Tim Diels
@timdiels
Apr 17 2017 16:11
I'd like to remove my databases on nextflow clean, seems that would be simplest and would be good enough
So, I'd like to run something for each value in a channel on cleanup
Paolo Di Tommaso
@pditommaso
Apr 17 2017 16:11
if your database is stored in a task work-dir, it will be removed
Tim Diels
@timdiels
Apr 17 2017 16:12
It's a decypher database, need to remove it by running a command
It's not part of a work-dir or even accessible through the filesystem
Paolo Di Tommaso
@pditommaso
Apr 17 2017 16:13
I see, so it cannot be removed by nextflow clean
you will need a custom script
Tim Diels
@timdiels
Apr 17 2017 16:13
Okay, then I'll let the pipeline generate a cleanup script
Paolo Di Tommaso
@pditommaso
Apr 17 2017 16:13
oh! smart .. :)
Tim Diels
@timdiels
Apr 17 2017 16:14
Maybe even a nextflow script :)
Tim Diels
@timdiels
Apr 17 2017 16:48
process1 --data--> process2. When invalidating cache of process1, will it rerun on the next -resume? (I'd guess so; on second thought it'd be an error if it didn't)
Paolo Di Tommaso
@pditommaso
Apr 17 2017 16:48
yes
it does
Tim Diels
@timdiels
Apr 17 2017 16:49
Makes sense, thanks
Shawn Rynearson
@srynobio
Apr 17 2017 18:07
Using the SLURM executor (or nextflow) is their a way to launch jobs to multiple node systems. Currently I only see on using the clusterOptions.
process {
    executor='slurm'
    clusterOptions='-A myNodes -p node-guest'
}
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:10
that's supposed to be the default behaviour
tip: you should use queue instead of -p option eg
process {
    executor='slurm'
    queue='node-guest'
    clusterOptions='-A myNodes'
}
Tim Diels
@timdiels
Apr 17 2017 18:13
When the workflow is cancelled, how are scripts terminated? Do they receive SIGTERM perhaps?
Shawn Rynearson
@srynobio
Apr 17 2017 18:13
Okay thanks! I understand the option default. I was just wondering if I could tell nextflow to run on multiple node systems.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:13
with the scancel command
I was just wondering if I could tell nextflow to run on multiple node systems.
NF just submits the job executions with sbatch, thus the question is: How SLURM executed jobs on multiple node systems
does it make sense ?
Shawn Rynearson
@srynobio
Apr 17 2017 18:16
The reason I ask is because on our HPC we have dedicated nodes, but they allow users to utilized idle (which have different accounts, etc) nodes as well. I wanted to check if nextflow allows this to happen somehow.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:17
there's no special NF way to handle the job execution
how do you manage this use case when submitting the jobs with sbatch ?
Shawn Rynearson
@srynobio
Apr 17 2017 18:18
Okay got it. I just wanted to check if the option was there.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:19
@timdiels How scripts are terminated depends by the executor you are using
when using the local executor, they are killed with a SIGTERM signal
Shawn Rynearson
@srynobio
Apr 17 2017 18:20
I've created a script that checks the sacctmgr information, and allows sbatch jobs to be launched to any idle node available.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:21
in practice how is the sbatch command line ?
Shawn Rynearson
@srynobio
Apr 17 2017 18:26

I collect the user node access with sacctmgr which also collects all the nodes information (cpus, memory, etc) then I can launch to any node using the -M option in sbatch.

Something like:
sbatch test.sbatch -A guest-nodes -p cluster-nodes -M myCluster

This is good for saturation of a HPC environment, above any dedicated nodes.

Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:28
not sure to understand, supposing you want to submit n jobs
for each job you execute sacctmgr to find the node where to submit it ?
Shawn Rynearson
@srynobio
Apr 17 2017 18:36
No I just execute that command as need be. Search for ican_find here to maybe better explain how I do it.
However as far as nextflow goes, I was just looking for multiple cluster launch. I think I can look for script-able away around this.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:37
I see, anyhow if you are using sbatch test.sbatch -A guest-nodes -p cluster-nodes -M myCluster
you can do that same setting clusterOptions = '-A guest-nodes -p cluster-nodes -M myCluster'
Shawn Rynearson
@srynobio
Apr 17 2017 18:39
That a was the path I was considering going down. Simple scripting in the clusterOption line of the config file.
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:39
:+1:
Shawn Rynearson
@srynobio
Apr 17 2017 18:40
Thanks @pditommaso
Paolo Di Tommaso
@pditommaso
Apr 17 2017 18:40
welcome
Tim Diels
@timdiels
Apr 17 2017 20:11
@pditommaso Does the local executor send SIGKILL after a while? If so, how long does it wait?
Paolo Di Tommaso
@pditommaso
Apr 17 2017 20:18
no, as soon as you abort the execution with CTRL+C
Tim Diels
@timdiels
Apr 17 2017 23:51
My signal handler does not fully execute when cancelling nextflow. This does not occur when running the script outside of nextflow (paste script body into a file and this is the output https://pastebin.com/x7jrLQVg). To reproduce: main.nf: https://pastebin.com/9iWbekmd My output: https://pastebin.com/zxhSXvzj