These are chat archives for nextflow-io/nextflow

21st
Mar 2019
Hugues Fontenelle
@huguesfontenelle
Mar 21 09:04

Hei! What does this mean"

 Script base path does not exist or is not a directory: /Users/hugues/work/ngs-pipeline/pipelines/bin

What am I supposed to put in bin?

Paolo Di Tommaso
@pditommaso
Mar 21 09:07
little, just a debugging info
Hugues Fontenelle
@huguesfontenelle
Mar 21 09:08
I have ignored it for the past few years. But I can't live in ignorance anymore :D
Paolo Di Tommaso
@pditommaso
Mar 21 09:09
I'm sure you are already putting your executable scripts in that folder
Hugues Fontenelle
@huguesfontenelle
Mar 21 09:10
another bin, which I add to the PATH.
OK thanks!
Paolo Di Tommaso
@pditommaso
Mar 21 09:12
putting there, NF automatically adds it to the PATH
Hugues Fontenelle
@huguesfontenelle
Mar 21 09:27
I've nested the pipeline.nf in a subdir called pipelines so the pipelines/bin location becomes un-intuitive. But this isn't important.
Hugues Fontenelle
@huguesfontenelle
Mar 21 09:41

OK, before I make another bug report (that would be a duplicate):

when I launch a pipeline using google-genomics with --outDir gs://my-google-bucket/work it works.
If I launch it with --out-dir gs://my-google-bucket/work it does not write anything to the output bucket.
Somehow the hyphenated resolution to camelCase (eg.: CLI --out-dir to script params.outDir) does not work for google-genomics.

Yasset Perez-Riverol
@ypriverol
Mar 21 10:26
Hi when using containers in nextflow can we have a case of one process -> multiple containers? menaing can we use multiple containers in the same process?
Paolo Di Tommaso
@pditommaso
Mar 21 10:27
one process => many task
one task => one container
Yasset Perez-Riverol
@ypriverol
Mar 21 10:28
the in principal you can have:
Evan Floden
@evanfloden
Mar 21 10:28
So yes. container is a directive and directives can be task specific.
Paolo Di Tommaso
@pditommaso
Mar 21 10:29
but I think yasset wants to use more than one container in the same task
Yasset Perez-Riverol
@ypriverol
Mar 21 10:29
process A {
 container  ’namespace/container1’
 container ’namespace/container2'
}
Paolo Di Tommaso
@pditommaso
Mar 21 10:29
nope
Yasset Perez-Riverol
@ypriverol
Mar 21 10:29
I just want to know if that is possible
ok them is one process => one container
Maxime Garcia
@MaxUlysse
Mar 21 10:29
Can't you put multiple tool in one container?
Yasset Perez-Riverol
@ypriverol
Mar 21 10:30
yes we can do it
Paolo Di Tommaso
@pditommaso
Mar 21 10:30
ok them is one process => one container
Yasset Perez-Riverol
@ypriverol
Mar 21 10:30
then, the container should encapsulate the multiple tools
Paolo Di Tommaso
@pditommaso
Mar 21 10:30
it's slightly different, the container can be evaluated at runtime, therefore each task execution can use a different one
but not more than one at time
Evan Floden
@evanfloden
Mar 21 10:32
If you see this example, the container is defined by a variable. So one process -> different containers. One task -> one container.
Yasset Perez-Riverol
@ypriverol
Mar 21 10:33
ok
micans
@micans
Mar 21 10:36
@rsuchecki @lebernstein this was my take:
a = Channel.from(['a', 'b', 'c', 'd', 'e', 'f', 'g'])

found = false
a.filter{ found || (it == 'd' && (found = true)) }
.view()
Yasset Perez-Riverol
@ypriverol
Mar 21 10:41
@evanfloden @pditommaso what do you think is the best way then to extract from a nextflow definition which are the containers used in the workflow
do you have any code around that allow use to interpret the execution of a nextflow to detect which containers has been used
Paolo Di Tommaso
@pditommaso
Mar 21 10:43
best practice is to put in the config, then nextflow config will tell you
Yasset Perez-Riverol
@ypriverol
Mar 21 10:44
in the example provided by @evanfloden is actually a parameter
Paolo Di Tommaso
@pditommaso
Mar 21 10:45
then it's not a best practice :satisfied:
Kevin Sayers
@KevinSayers
Mar 21 10:48
@ypriverol you can use traceand get a list of all the containers used in a workflow too
Yasset Perez-Riverol
@ypriverol
Mar 21 10:50
yes, but trace will need to execute the workflow
Tim Dudgeon
@tdudgeon
Mar 21 11:25
@pditommaso I don't think that TraceFileObserver will help me here? It would just let me write the fields that are available to the trace report in a different way?
What I'm needing is to write my own information (workflow specific) to a log file that is present in the workflow execution directory e.g. like the trace.txt or .nextflow.log but I define what goes in this file.
Paolo Di Tommaso
@pditommaso
Mar 21 11:32
it depends what kind of events you are trying to capture
Tim Dudgeon
@tdudgeon
Mar 21 11:55
I'm wanting to write information that would be fed back to the user in (almost) real time that indicates how the workflow is progressing. And that information is specific to the particular workflow. As each task completes I would have something to report e.g. a message and/or a measure of the progress.
Alexander Peltzer
@apeltzer
Mar 21 11:57
weblog feature? ;-)
Tim Dudgeon
@tdudgeon
Mar 21 12:01
no, because that is again restricted to the information that can be reported in the trace file. I can't send custom data.
I'm thinking I might be able to do some channel trickery here. I'll let you know if I mange it!
Alexander Peltzer
@apeltzer
Mar 21 12:02
log.info might help ?
This is shown on the general nextflow execution "log" for nf-core/rnaseq
So users will e.g. know when a sample doesn't yield more than 5% aligned reads
Paolo Di Tommaso
@pditommaso
Mar 21 12:39
Tim Dudgeon
@tdudgeon
Mar 21 17:09
I got something working for the status reporting by writing status info to channels and then combining those channels and writing to a file.
https://gist.github.com/tdudgeon/8436f11603884906dab5647818e5aa1a#file-main-nf
If anyone sees a better way to do this then let me know!
Félix C. Morency
@fmorency
Mar 21 20:01
@pditommaso I have a DataflowVariable that is a flattened list of string and I want to manipulate this list in Groovy. Is there a clean way of doing so? I want to do something like
input:
val(parametric_maps) from Channel.value( ['a', 'b', 'c'] )

script:
if ("a" in ${parametric_maps}) { ... } // DOESN'T WORK
"""
"""
Olga Botvinnik
@olgabot
Mar 21 20:27
Hellow! How can one specify the NCBI API key for Channel.fromSRA? I saved a file as ~/.ncbi but am not sure how to specify it in the script
 Wed 20 Mar - 22:19~/code/nf-kmer-similarity/from-csvs   origin ☊ master ↑12☀ 
  make run_aws
nextflow run main.nf \
        -work-dir s3://olgabot-maca/nextflow-workdir-test/ \
        -bucket-dir s3://olgabot-maca/nextflow-bucket-dir-test/ \
        -with-trace -with-timeline -with-dag -with-report -latest -resume
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:pom:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-nio:pom:2.5.6
CAPSULE: Downloading dependency io.nextflow:nextflow:pom:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy:pom:2.5.6
CAPSULE: Downloading dependency io.nextflow:nf-commons:pom:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-json:pom:2.5.6
CAPSULE: Downloading dependency com.jcraft:jzlib:pom:1.1.1
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-templates:pom:2.5.6
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-xml:pom:2.5.6
CAPSULE: Downloading dependency org.eclipse.jgit:org.eclipse.jgit:pom:5.2.1.201812262042-r
CAPSULE: Downloading dependency org.eclipse.jgit:org.eclipse.jgit-parent:pom:5.2.1.201812262042-r
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:jar:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-nio:jar:2.5.6
CAPSULE: Downloading dependency io.nextflow:nextflow:jar:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy:jar:2.5.6
CAPSULE: Downloading dependency io.nextflow:nf-commons:jar:19.03.0-edge
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-json:jar:2.5.6
CAPSULE: Downloading dependency com.jcraft:jzlib:jar:1.1.1
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-templates:jar:2.5.6
CAPSULE: Downloading dependency org.codehaus.groovy:groovy-xml:jar:2.5.6
CAPSULE: Downloading dependency org.eclipse.jgit:org.eclipse.jgit:jar:5.2.1.201812262042-r
N E X T F L O W  ~  version 19.03.0-edge
Launching `main.nf` [high_noether] - revision: ff060a2f56
WARN: Missing NCBI API key -- Read more https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/
Olga Botvinnik
@olgabot
Mar 21 20:35
Figured it out (from https://github.com/nextflow-io/nextflow/blob/40044dea37f6dc4ca21507ded1e7d0c0ca6505e6/modules/nextflow/src/main/groovy/nextflow/datasource/SraExplorer.groovy)-- you need to set NXF_NCBI_APIKEY:
 ✘  Thu 21 Mar - 16:33  ~/code/nf-kmer-similarity/from-csvs   origin ☊ master ↑1 55● 
  export NXF_NCBI_APIKEY=redacted
(base)
 Thu 21 Mar - 16:34  ~/code/nf-kmer-similarity/from-csvs   origin ☊ master ↑1 55● 
  make run_aws
nextflow run main.nf \
        -work-dir s3://olgabot-maca/nextflow-workdir-test/ \
        -bucket-dir s3://olgabot-maca/nextflow-bucket-dir-test/ \
        -with-trace -with-timeline -with-dag -with-report -latest -resume
N E X T F L O W  ~  version 19.03.0-edge
Launching `main.nf` [amazing_lamarr] - revision: ff060a2f56
Tobias "Tobi" Schraink
@tobsecret
Mar 21 20:36
@olgabot that's probably something that's worth mentioning in the docs
Olga Botvinnik
@olgabot
Mar 21 21:53
@tobsecret happy to contribute! I couldn't get the ncbi.apiKey to work, e.g. adding -e.ncbi.apiKey=redacted to the command line didn't work. How can one specify this?
Tobias "Tobi" Schraink
@tobsecret
Mar 21 21:56
@olgabot
I have no clue - I'm no dev :sweat_smile:
I just know that if I ran into that error and it wasn't in the docs, I probably wouldn't be able to just pick it up from the source like you did
Olga Botvinnik
@olgabot
Mar 21 21:56
It semi-was in the docs but without an example so I missed it
Tobias "Tobi" Schraink
@tobsecret
Mar 21 21:59
Oooh, then it's a great extension for the FAQ.
You can simply write the question/answer pair here: https://github.com/nextflow-io/website/blob/faq/content/faq.md
@pditommaso mentioned in this thread: https://github.com/nextflow-io/website/issues/14#issuecomment-474001170 that this is the way to contribute to the FAQ
I'm trying to collect a bunch of FAQ questions that people have, so if you don't want to make your own PR, you can just send it to me here and I'll add it to the 3 other questions I've collected