These are chat archives for nextflow-io/nextflow

12th
Sep 2017
spaceturtle
@spaceturtle
Sep 12 2017 09:18
Can we set two different "publishDir" for one process? Each of them store one specific types of files?
Phil Ewels
@ewels
Sep 12 2017 10:13
Yup! If you see our pipeline at the above link you can see that each process saves its files in a different subfolder
But you can use totally different paths for each process if you want. You just need to supply a file path.
Steve Marshall
@stevemmarshall
Sep 12 2017 15:19
@pditommaso I was looking at this video https://asciinema.org/a/9vupd4d72ivaz6h56pajjjkop and one thing I didn't see is that do you need to run the nextflow daemon on all the nodes in the cluster? When I configured my version and created a mini cluster, it ran locally rather than on the cluster..
Paolo Di Tommaso
@pditommaso
Sep 12 2017 15:21
the command nextflow cloud create setup the cloud environment for you
Steve Marshall
@stevemmarshall
Sep 12 2017 15:24
ok that's what I did... I was trying one of the blast examples with multiple fasta files but it seemed to be running locally
Paolo Di Tommaso
@pditommaso
Sep 12 2017 15:24
do you have the execution stdout ?
Steve Marshall
@stevemmarshall
Sep 12 2017 15:24
ya, hold on
Steve Marshall
@stevemmarshall
Sep 12 2017 15:42
quick question, in terms of security when creating the cluster what is the best way for logging into the master and then running jobs
Paolo Di Tommaso
@pditommaso
Sep 12 2017 15:43
ssh, should be secure enough
Steve Marshall
@stevemmarshall
Sep 12 2017 16:38
Hi @pditommaso so I ran this command nextflow run test/examples/multipleinputs.nf --query test/examples/data/pro t*.fa
and got this output... N E X T F L O W ~ version 0.25.1
Launching test/examples/multiple_inputs.nf [amazing_heisenberg] - revision: 5a8658d396
[warm up] executor > local
[d9/b4623d] Submitted process > blast (prot_2)
[49/aa0d5c] Submitted process > blast (prot_5)
[f9/eb2be2] Submitted process > blast (prot_1)
[82/02e2ec] Submitted process > blast (prot_4)
[19/ae20e9] Submitted process > blast (prot_3)
Result saved to file: /home/ubuntu/blast_result.txt
I ran the command on the master node
Steve Marshall
@stevemmarshall
Sep 12 2017 18:19
When I ran it like this nextflow run test/examples/multipleinputs.nf -process.executor ignite --query test/examples/data/prot*.fa
it returned this... it seems to be taking slightly longer and looking at the log output, it doesn't seem like it submitted a job to the other node.. N E X T F L O W ~ version 0.25.1
Launching test/examples/multiple_inputs.nf [stupefied_jang] - revision: 48c2ae88cd
[warm up] executor > ignite
[4e/3c9ab4] Submitted process > blast (prot_1)
[d4/125e99] Submitted process > blast (prot_4)
[ee/1613f5] Submitted process > blast (prot_5)
[23/190175] Submitted process > blast (prot_2)
[21/cf12e3] Submitted process > blast (prot_3)
Result saved to file: /home/ubuntu/blast_result.txt
Paolo Di Tommaso
@pditommaso
Sep 12 2017 19:44
the first is running locally, you can recognise by the line executor > local
the second is distributed executor > ignite
but you would not need to specify the ignite executor on the command
are you launching those in the cloud master node?
Venkat Malladi
@vsmalladi
Sep 12 2017 20:16
Hellow
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:17
Hellow ..
Venkat Malladi
@vsmalladi
Sep 12 2017 20:17
Can anyone point me to a good example to nextflow configuration that would run on a HPC using SLURM and docker
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:18
on-premises or cloud ?
Venkat Malladi
@vsmalladi
Sep 12 2017 20:18
on premises
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:20
ok, as for NF configuration you should define something like
process.executor = 'slurm'
process.container = 'your/image'
docker.enabled = true
done
provided you have docker installed in your cluster ..
said that on HPC clusters I strongly advice to use Singularity instead of Docker
Venkat Malladi
@vsmalladi
Sep 12 2017 20:24
Correct that is the way we are going
now I can set up another NF configuration file that would allow another person to load just using docker images
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:25
you can do that is a separate config file
or by using config profiles
Venkat Malladi
@vsmalladi
Sep 12 2017 20:26
The profiles looks like more of what I want
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:26
:+1:
Venkat Malladi
@vsmalladi
Sep 12 2017 20:29
If I am allowing for this flexibility and I need to load modules (e.g module load samtools) for a given process on my cluster, but want that process to work for the person using docker how to you recommend setting up my .nf file
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:31
I would create two separate config profiles, the standard ie default using modules and another enabling docker
Venkat Malladi
@vsmalladi
Sep 12 2017 20:39
Sorry I am not sure I fully understand, can you elaborate
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:40
you can create a config with two profiles eg
Venkat Malladi
@vsmalladi
Sep 12 2017 20:41
so for each process I can specific a specific container
Paolo Di Tommaso
@pditommaso
Sep 12 2017 20:41
profiles { 
  standard {
    process.module = 'foo/bar'
    process.executor = 'slurm'
  }

  runWitHDocker {
    process.container = 'image/name'
    docker.enabled = true
  }
}
eventually yes
I tend to use a single fat container
Venkat Malladi
@vsmalladi
Sep 12 2017 20:53
okay
Steve Marshall
@stevemmarshall
Sep 12 2017 21:31
@pditommaso yes I'm running that on the master node
Paolo Di Tommaso
@pditommaso
Sep 12 2017 21:32
looks weird, it seems something is missing in the config
what AMI are you using ?
Steve Marshall
@stevemmarshall
Sep 12 2017 21:35
I made a custom ami with Ubuntu 16.04
Paolo Di Tommaso
@pditommaso
Sep 12 2017 21:37
are set env variables starting with NXF_ ?
Steve Marshall
@stevemmarshall
Sep 12 2017 21:39
how do you set/dump all the env vars?
Paolo Di Tommaso
@pditommaso
Sep 12 2017 21:39
env | grep NXF
Venkat Malladi
@vsmalladi
Sep 12 2017 21:54
Is there variable for the process name?
process qc_fastq {
tag "$name"
publishDir "$baseDir/output/$name/qc_fastq", mode: 'copy'
Paolo Di Tommaso
@pditommaso
Sep 12 2017 21:55
task.process
Venkat Malladi
@vsmalladi
Sep 12 2017 21:55
thanks
Paolo Di Tommaso
@pditommaso
Sep 12 2017 21:55
welcome
Steve Marshall
@stevemmarshall
Sep 12 2017 22:27
so i have no NXF vars