These are chat archives for nextflow-io/nextflow

21st
Mar 2017
Evan Floden
@evanfloden
Mar 21 2017 08:59

@sridhar0605 When you execute:
nextflow run skptic/tuxedo-nf
for the first time, NF pulls the docker image.
In the file params_default.configyou can see that this would be equal to running:

docker pull cbcrg/tuxedo-nf@sha256:e9bc1b3d648431591c328ee0dfd297b5315f9bef659f377606ff0fbfbd81c5e5

before running the pipeline. The sha256 hash just ensures you run the exact docker image (for reproducibility). I have just checked the docker pull command and it is working from my end.

Roman Valls Guimera
@brainstorm
Mar 21 2017 09:28
How come nextflow is not detecting my cluster profile when indicated to run with it in the cmdline?: http://pastebin.com/e1FbyC9B
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:29
config profiles must be defined in the profiles scope ..
Roman Valls Guimera
@brainstorm
Mar 21 2017 09:35
Ugh, misread and thought they were global, thanks!!
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:36
welcome
Roman Valls Guimera
@brainstorm
Mar 21 2017 09:37
Now fingers crossed it works on the proprietary PBS Pro (I've been told it's different from PBS Torque)
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:38
let's see, yes there are many different distribution with slight differences
Paolo Di Tommaso
@pditommaso
Mar 21 2017 12:34
Copy
Tim Diels
@timdiels
Mar 21 2017 12:35
Staging out (with scratch enabled) is the copying/moving of output to the cache?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 13:09
@timdiels yes
well, is the copy from the local scratch into the shared pipeline work dir
Tim Diels
@timdiels
Mar 21 2017 13:27
And the work dir is just where the input/output normally would have been if scratch false?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 13:27
yes
Tim Diels
@timdiels
Mar 21 2017 15:09
is there a difference between Channel.create() and Channel.value()?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 15:09
the real question should be: what's the difference between Channel.from() and Channel.value() :)
Channel.create() creates a dataflow queue
Channel.value() creates a dataflow value
the main difference is that to the first you can bind many values
the send you can bind only one value
Tim Diels
@timdiels
Mar 21 2017 15:55
Ok, thanks
https://www.nextflow.io/docs/latest/operator.html#filter
"that returns only string that begins with a"
It only returns strings consisting of all a's, e.g. ab is not returned
Roman Valls Guimera
@brainstorm
Mar 21 2017 15:58
Basic question: is it possible to expose the params scope on the cmdline? I.e I have params.threads defined in the main.nf and I want to override it with another value in the cli?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:03
It only returns strings consisting of all a's, e.g. ab is not returned
you are right, thanks for pointing it out
I want to override it with another value in the cli
maybe you want to do something like this
params.foo = params.bar
?
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:25
More something like this nextflow run main.nf -with-timeline timeline.html -with-trace -with-dag flowchart.png -profile cluster -params.threads = 16 when in the main.nf it is defined by default as params.threads = 1
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:33
you only need to replace -params.threads with --threads 16 on the command line
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:46
Sweet!
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:50
:v:
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:52
Moving on to kubernetes now :)
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:53
enjoy !
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:38
Hmm, the container keyword seems to be mandatory for both process.executor='k8s' scope and main.nf's process, which kind of breaks my idea to run the same .nf in different environments (aws, hpc & k8s) :/... ideally the container shouldn't be needed for the process if defined in the kubernetes profile, right?:
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile kubernetes
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [hungry_pasteur] - revision: 189770e423
[warm up] executor > k8s
Error executing process > 'align (1)'

Caused by:
  Missing container for process `align`
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:42
umm .. kubernetes is meant to run containers, isn't it?
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:42
Yes, I would like to run my workflow in the same container
bcbio
And actually from another provider than dockerhub in the future: https://quay.io/repository/biocontainers/bcbio-nextgen?tab=tags
Sorry, I'm messing up the use cases it was designed for :-S
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:43
NF run a pod for each process in your pipeline when you use the k8s executor
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:44
Ok, each of those pods can be mapped to the same container, no?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:44
yes of course
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:45
So, I don't want to define container in my workflow (main.nf), because there's no docker in my HPC
So I'll have to change/comment that definition when switching envs... or create a specific (copied) workflow for k8s :/
Changing -profile in the cmdline should select the environment, without modifying the workflow itself... hope I'm explaining myself :-S
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:54
not so much .. ;)
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:55
Ouch XD
Ok, I don't want to touch main.nf, kind of like java in the old times, write once, run everywhere, right?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:55
however I would suggest you to give a look to this posts
and
I don't want to touch main.nf, kind of like java in the old times, write once, run everywhere, right?
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:56
Then, in my mind, I should be able to define different compute environments/runners in netflow.config and then launch to those via -profile
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:56
yes, the config file is meant for that
exactly
or different config files and then specify on the cmd line
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:57
Ok, then, this happens and ruins my dream:
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile standard
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [condescending_noether] - revision: 189770e423
[warm up] executor > local
[4e/4b210a] Submitted process > align (1)
[1c/bdeede] Submitted process > sort (1)
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile kubernetes
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [lonely_mcclintock] - revision: 189770e423
[warm up] executor > k8s
Error executing process > 'align (1)'

Caused by:
  Missing container for process `align`
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:58
you need to add in your config/profile
process.container = 'container/name'
Roman Valls Guimera
@brainstorm
Mar 21 2017 18:00
Yes, like this?:
    kubernetes {
        process.executor = 'k8s'
        cpus = 1
        memory = '1 GB'
        container = 'bcbio/bcbio:latest'
        /* container = "quay.io/biocontainers/bcbio-nextgen:1.0.1--py27_0" */
    }
Paolo Di Tommaso
@pditommaso
Mar 21 2017 18:01
nope
like this
profiles {
 kubernetes {
        process.executor = 'k8s'
        process.cpus = 1
        process.memory = '1 GB'
        process.container = 'bcbio/bcbio:latest'

    }
}
Roman Valls Guimera
@brainstorm
Mar 21 2017 18:06
Ahaaa
Thx!
Seems to work now, but there's some fishy Route53/DNS/certificates issue going on:
Command output:
  Unable to connect to the server: x509: certificate is valid for pmos.paulmunnsinstantlawn.com.au, not api.cluster.umccr.nopcode.org
Paolo Di Tommaso
@pditommaso
Mar 21 2017 18:11
Enjoy kubernetes!