These are chat archives for nextflow-io/nextflow

21st
Mar 2017
Evan Floden
@evanfloden
Mar 21 2017 08:59 UTC

@sridhar0605 When you execute:
nextflow run skptic/tuxedo-nf
for the first time, NF pulls the docker image.
In the file params_default.configyou can see that this would be equal to running:

docker pull cbcrg/tuxedo-nf@sha256:e9bc1b3d648431591c328ee0dfd297b5315f9bef659f377606ff0fbfbd81c5e5

before running the pipeline. The sha256 hash just ensures you run the exact docker image (for reproducibility). I have just checked the docker pull command and it is working from my end.

Roman Valls Guimera
@brainstorm
Mar 21 2017 09:28 UTC
How come nextflow is not detecting my cluster profile when indicated to run with it in the cmdline?: http://pastebin.com/e1FbyC9B
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:29 UTC
config profiles must be defined in the profiles scope ..
Roman Valls Guimera
@brainstorm
Mar 21 2017 09:35 UTC
Ugh, misread and thought they were global, thanks!!
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:36 UTC
welcome
Roman Valls Guimera
@brainstorm
Mar 21 2017 09:37 UTC
Now fingers crossed it works on the proprietary PBS Pro (I've been told it's different from PBS Torque)
Paolo Di Tommaso
@pditommaso
Mar 21 2017 09:38 UTC
let's see, yes there are many different distribution with slight differences
Paolo Di Tommaso
@pditommaso
Mar 21 2017 12:34 UTC
Copy
Tim Diels
@timdiels
Mar 21 2017 12:35 UTC
Staging out (with scratch enabled) is the copying/moving of output to the cache?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 13:09 UTC
@timdiels yes
well, is the copy from the local scratch into the shared pipeline work dir
Tim Diels
@timdiels
Mar 21 2017 13:27 UTC
And the work dir is just where the input/output normally would have been if scratch false?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 13:27 UTC
yes
Tim Diels
@timdiels
Mar 21 2017 15:09 UTC
is there a difference between Channel.create() and Channel.value()?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 15:09 UTC
the real question should be: what's the difference between Channel.from() and Channel.value() :)
Channel.create() creates a dataflow queue
Channel.value() creates a dataflow value
the main difference is that to the first you can bind many values
the send you can bind only one value
Tim Diels
@timdiels
Mar 21 2017 15:55 UTC
Ok, thanks
https://www.nextflow.io/docs/latest/operator.html#filter
"that returns only string that begins with a"
It only returns strings consisting of all a's, e.g. ab is not returned
Roman Valls Guimera
@brainstorm
Mar 21 2017 15:58 UTC
Basic question: is it possible to expose the params scope on the cmdline? I.e I have params.threads defined in the main.nf and I want to override it with another value in the cli?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:03 UTC
It only returns strings consisting of all a's, e.g. ab is not returned
you are right, thanks for pointing it out
I want to override it with another value in the cli
maybe you want to do something like this
params.foo = params.bar
?
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:25 UTC
More something like this nextflow run main.nf -with-timeline timeline.html -with-trace -with-dag flowchart.png -profile cluster -params.threads = 16 when in the main.nf it is defined by default as params.threads = 1
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:33 UTC
you only need to replace -params.threads with --threads 16 on the command line
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:46 UTC
Sweet!
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:50 UTC
:v:
Roman Valls Guimera
@brainstorm
Mar 21 2017 16:52 UTC
Moving on to kubernetes now :)
Paolo Di Tommaso
@pditommaso
Mar 21 2017 16:53 UTC
enjoy !
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:38 UTC
Hmm, the container keyword seems to be mandatory for both process.executor='k8s' scope and main.nf's process, which kind of breaks my idea to run the same .nf in different environments (aws, hpc & k8s) :/... ideally the container shouldn't be needed for the process if defined in the kubernetes profile, right?:
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile kubernetes
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [hungry_pasteur] - revision: 189770e423
[warm up] executor > k8s
Error executing process > 'align (1)'

Caused by:
  Missing container for process `align`
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:42 UTC
umm .. kubernetes is meant to run containers, isn't it?
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:42 UTC
Yes, I would like to run my workflow in the same container
bcbio
And actually from another provider than dockerhub in the future: https://quay.io/repository/biocontainers/bcbio-nextgen?tab=tags
Sorry, I'm messing up the use cases it was designed for :-S
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:43 UTC
NF run a pod for each process in your pipeline when you use the k8s executor
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:44 UTC
Ok, each of those pods can be mapped to the same container, no?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:44 UTC
yes of course
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:45 UTC
So, I don't want to define container in my workflow (main.nf), because there's no docker in my HPC
So I'll have to change/comment that definition when switching envs... or create a specific (copied) workflow for k8s :/
Changing -profile in the cmdline should select the environment, without modifying the workflow itself... hope I'm explaining myself :-S
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:54 UTC
not so much .. ;)
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:55 UTC
Ouch XD
Ok, I don't want to touch main.nf, kind of like java in the old times, write once, run everywhere, right?
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:55 UTC
however I would suggest you to give a look to this posts
and
I don't want to touch main.nf, kind of like java in the old times, write once, run everywhere, right?
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:56 UTC
Then, in my mind, I should be able to define different compute environments/runners in netflow.config and then launch to those via -profile
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:56 UTC
yes, the config file is meant for that
exactly
or different config files and then specify on the cmd line
Roman Valls Guimera
@brainstorm
Mar 21 2017 17:57 UTC
Ok, then, this happens and ruins my dream:
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile standard
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [condescending_noether] - revision: 189770e423
[warm up] executor > local
[4e/4b210a] Submitted process > align (1)
[1c/bdeede] Submitted process > sort (1)
(py2) romanvg@romanvg:~/dev/bwa-nextflow-benchmark$ nextflow run main.nf -profile kubernetes
N E X T F L O W  ~  version 0.23.4
Launching `main.nf` [lonely_mcclintock] - revision: 189770e423
[warm up] executor > k8s
Error executing process > 'align (1)'

Caused by:
  Missing container for process `align`
Paolo Di Tommaso
@pditommaso
Mar 21 2017 17:58 UTC
you need to add in your config/profile
process.container = 'container/name'
Roman Valls Guimera
@brainstorm
Mar 21 2017 18:00 UTC
Yes, like this?:
    kubernetes {
        process.executor = 'k8s'
        cpus = 1
        memory = '1 GB'
        container = 'bcbio/bcbio:latest'
        /* container = "quay.io/biocontainers/bcbio-nextgen:1.0.1--py27_0" */
    }
Paolo Di Tommaso
@pditommaso
Mar 21 2017 18:01 UTC
nope
like this
profiles {
 kubernetes {
        process.executor = 'k8s'
        process.cpus = 1
        process.memory = '1 GB'
        process.container = 'bcbio/bcbio:latest'

    }
}
Roman Valls Guimera
@brainstorm
Mar 21 2017 18:06 UTC
Ahaaa
Thx!
Seems to work now, but there's some fishy Route53/DNS/certificates issue going on:
Command output:
  Unable to connect to the server: x509: certificate is valid for pmos.paulmunnsinstantlawn.com.au, not api.cluster.umccr.nopcode.org
Paolo Di Tommaso
@pditommaso
Mar 21 2017 18:11 UTC
Enjoy kubernetes!