These are chat archives for nextflow-io/nextflow

19th
Feb 2019
Laurence E. Bernstein
@lebernstein
Feb 19 01:56
This message was deleted

So I have the following trimming process which outputs both input fastq files for bwa:

read_pair = "XXX_R{1,2}_001.fastq.gz"

process '0AB_cut_fastq' {
  publishDir "${workingDir}", mode: 'copy'
  input:
    file referenceFile from ref_file
    inputPath = Channel.fromPath(read_pair)  // One string at a time
    file inputFastq from inputPath           // Conversion to file (2x)
  output:
    set "${inputFastq.baseName}_Trimmed.fastq" into trimmed_fast_ch
  script:
     """
      ${cutFastqDir}/CutFastq.pl  \
      ${trimLength}                                \
      ${inputFastq}               \
      1> "${inputFastq.baseName}_Trimmed.fastq" 
    """
}

And after that I would like to use BOTH trimmed files (as forward and reverse reads) as inputs to a single BWA process:

process '1A_bwa_align' {
  publishDir "${workingDir}", mode: 'copy'
  input:
    file referenceFile from ref_file
    //file trimmedFastq from trimmed_fast_ch  -- This would run the process twice
    //set forward, reverse from trimmed_fast_ch  -- This yields null for reverse
  output:
  script:
  """
    ${bwaBinDir}/${bwaFlags}    \
      ${referenceFile}              \
      ${forward}  ${reverse}              \
      1> "${sampleName}".Aligned.sam
  """
}
Hugues Fontenelle
@huguesfontenelle
Feb 19 06:57

@lebernstein Have you checked https://www.nextflow.io/docs/latest/operator.html#tolist ?

file trimmedFastq from trimmed_fast_ch.toList()

Then access the files with an index as intrimmedFastq[0].

@pinin4fjords

nextflow run https://github.com/my/repo/pipeline1.nf

should work. With a /

Hugues Fontenelle
@huguesfontenelle
Feb 19 07:10

@pinin4fjords

Hi- can I supply lists via the Nextflow config? If not, what's the simplest workaround- e.g. to convert a comma-separated string to numeric list? (sorry, my Groovy Googling is failing me)

What have you tried? I have no problem with lists in config...

Qi ZHAO
@likelet
Feb 19 07:56
@pditommaso hi Paolo, dose the latest version of nextflow not support the duplicate process name in ifelse sentences? if so, which version was the feature add?
Maxime Garcia
@MaxUlysse
Feb 19 08:28
@pinin4fjords Have you tried ?:
# At the command line
nextflow run --foo bar,bar2,bar3
# In your code
foo = params.foo ? params.foo.split(',').collect{it.trim()} : []
Jonathan Manning
@pinin4fjords
Feb 19 08:37
@huguesfontenelle @MaxUlysse I didn't see any examples in the docs, but yep, lists in the config files do work fine- "foo = [ 0.1, 0.3, 0.5, 0.7, 1, 2, 3, 4, 5 ]". I should have just tried it- thanks.
Paolo Di Tommaso
@pditommaso
Feb 19 08:50
@likelet duplicate process names was a mistake since the beginning and it has been retired to enable modules and subworkflows a in future release
Qi ZHAO
@likelet
Feb 19 08:52
and even in a conditional sentence?
Qi ZHAO
@likelet
Feb 19 09:03
it works before but seems not allowed in the latest version.
so i have to tell users to update the code or the nextflow
Paolo Di Tommaso
@pditommaso
Feb 19 09:21
it's enough to give that processes two different names ..
Qi ZHAO
@likelet
Feb 19 09:23
emm, i did so to solve the errors.
Paolo Di Tommaso
@pditommaso
Feb 19 09:23
but ?
(provided there's a but)
Qi ZHAO
@likelet
Feb 19 09:25
but some of the users use the old code, i just want to notice them to know that the exact nextflow version still support this .
Paolo Di Tommaso
@pditommaso
Feb 19 09:27
version 18.10.1 still allows duplicate process names printing a warning message
Qi ZHAO
@likelet
Feb 19 09:27
:ok:
Paolo Di Tommaso
@pditommaso
Feb 19 09:27
:+1:
Qi ZHAO
@likelet
Feb 19 09:28
i will add an FAQ there. Thanks a lot
Paolo Di Tommaso
@pditommaso
Feb 19 09:28
welcome!
Bioninbo
@Bioninbo
Feb 19 10:20
Hello everyone
When I run my script I get the following error message: Command exit status: 1 Command output: (empty). However, the output file is created and the command works fine without error in the failing work folder. Do you know what could be the issue?
Yasset Perez-Riverol
@ypriverol
Feb 19 10:35
Hi guys, have you use before external libraries from Groovy inside a nextflow workflow?
Qi ZHAO
@likelet
Feb 19 10:37
i tried before, may have look of this project https://github.com/likelet/RNAseqPipe
@ypriverol
Yasset Perez-Riverol
@ypriverol
Feb 19 10:38
Thanks I will take a look
or just put the jar in the lib folder
Yasset Perez-Riverol
@ypriverol
Feb 19 10:39
cool.
Qi ZHAO
@likelet
Feb 19 10:39
cool
i like this feature
more like a java behavior
Paolo Di Tommaso
@pditommaso
Feb 19 10:39
@Bioninbo how do you know that the command works fine?
NF is groovy, groovy is java ..
Qi ZHAO
@likelet
Feb 19 10:40
ignore me .
:P
Bioninbo
@Bioninbo
Feb 19 10:41
@pditommaso when I run the .command.sh command in the failing work folder there is no error and the correct ouput is generated. Somehow it seems I fixed this error by adding "echo DONE" at the end of the bash script I am calling.
Paolo Di Tommaso
@pditommaso
Feb 19 10:42
what's the content of .exitcode file ?
Bioninbo
@Bioninbo
Feb 19 10:43
empty
Paolo Di Tommaso
@pditommaso
Feb 19 10:44
therefore something is failing
debug it running bash .command.run
Bioninbo
@Bioninbo
Feb 19 10:45
there is no error when I run this command
Paolo Di Tommaso
@pditommaso
Feb 19 10:46
and the content of .exitcode ?
Bioninbo
@Bioninbo
Feb 19 10:46
still empty
Anthony Ferrari
@af8
Feb 19 10:46

Is conditional publishDir directive still possible ?
publishDir path: {params.opt ? null : "${params.output_directory}" }, mode: 'copy', pattern: "${id}.vcf.{gz,gz.tbi}"

Nextflow (19.01) complains about the null with :

Caused by:
  Cannot invoke method complete() on null object
Paolo Di Tommaso
@pditommaso
Feb 19 10:47
it must contain zero, otherwise something is failing
Bioninbo
@Bioninbo
Feb 19 10:47
sorry the content is one
Paolo Di Tommaso
@pditommaso
Feb 19 10:48
@af8 the path cannot be null
Anthony Ferrari
@af8
Feb 19 10:50
OK I found the null alternative in a old gitter chat between you and phil ewels but it was in 2016
What is the solution to publish or not based on a boolean ?
Paolo Di Tommaso
@pditommaso
Feb 19 10:52
maybe it was a bug, anyhow is not a supported feature
Anthony Ferrari
@af8
Feb 19 10:53
OK, many thanks
Maxime Garcia
@MaxUlysse
Feb 19 10:58
@af8 Have you tried:
publishDir params.output_directory, saveAs : {
    if (it == "${id}.vcf.{gz,gz.tbi}" && params.opt) it
    else null
}
Yasset Perez-Riverol
@ypriverol
Feb 19 12:04
Do you have an exmaple about the use of http://docs.groovy-lang.org/latest/html/documentation/grape.html in nextflow
Paolo Di Tommaso
@pditommaso
Feb 19 12:21
is the same as groovy, put at top of the script
Anthony Ferrari
@af8
Feb 19 13:53
Thanks for the tip @MaxUlysse . It worked well !
Indeed I did not see that it was explained in the docs to use the null value in saveAs to prevent from publishing.
Can you explain the logic behind it == "${id}.vcf.{gz,gz.tbi}" part ?
Could we simply just rewrite this as :
publishDir params.output_directory, saveAs : {
    if (params.opt) "${id}.vcf.{gz,gz.tbi}"
    else null
}
Maxime Garcia
@MaxUlysse
Feb 19 13:54
@af8 the idea is to check if your output (it) match the pattern "${id}.vcf.{gz,gz.tbi}" to only catch that output
I'm not sure your option would work as you're specifying a glob pattern {gz,gz.tbi} as the output name
Stephen Kelly
@stevekm
Feb 19 14:00

@davidmasp

Is there a reason why nextflow is using source activate instead of conda activate?

sidenote: using conda activate assumes that conda is already in your PATH, something I have a lot of issues with when using conda because I keep many independent conda installations (one per project), also pretty sure I got source activate straight from the conda docs at some point, not sure the official Nextflow reason though. conda activate might be preferable because I have been having issues with source activate, it seems to be buggy sometimes under certain conditions.

Anthony Ferrari
@af8
Feb 19 15:32
@MaxUlysse for completeness, it ended up like this :
publishDir path:"${params.output_directory}", mode: 'copy', pattern: "${id}.vcf.{gz,gz.tbi}", saveAs: { if (!params.pass) it else null }
Maxime Garcia
@MaxUlysse
Feb 19 15:32
Very nice
I like it
lastwon1216
@lastwon1216
Feb 19 17:17
hi all,
is there anyway to get the part 'test' from test.fastq.gz from this param?
params.reads="$PWD/test.fastq.gz"
readsBase=file(params.reads).baseName
Paolo Di Tommaso
@pditommaso
Feb 19 17:21
params.reads="$PWD/test.fastq.gz"
readsBase=file(params.reads).simpleName
and .simpleName is the same as .getSimpleName()
lastwon1216
@lastwon1216
Feb 19 17:34
ah guess missed that part, when reading. thank you!
Clément ZOTTI
@czotti
Feb 19 18:04

I have a question regarding the clusterOptions (using SLURM), when I define it in a process like:

process A {
    clusterOptions '--exclude=node[01-04],node_test'
}

The generated .command.run contains the directive #SBATCH --exclude=node[01-04],node_test

But when I remove it from the process definition and pass it from the command line: nextflow run main.nf input -process.clusterOptions '--exclude=node[01-04],node_test'
The .command.run file do not contain the exclusion is it normal ?

Paolo Di Tommaso
@pditommaso
Feb 19 18:21
it should work, try to put in the config file
Clément ZOTTI
@czotti
Feb 19 18:32
It works when I put the exclude in the nextflow.config. But still removing it from the file and give it with the command line produce a .command.run without the exclude directive.
Nextflow version if it matters: nextflow version 0.30.2.4867
Stephen Kelly
@stevekm
Feb 19 18:33
when I run with -resume, its taking extremely long for Nextflow to finish, even if all tasks are cached. I am thinking this might have to do with copying files to the publishDir's. Is there a way to get nextflow to tell me in the .nextflow.log that it is doing so?
for example this one has been running all night with seemingly no progress update;
Feb-18 22:36:33.654 [Actor Thread 310] DEBUG nextflow.Session - <<< barrier arrive (process: custom_analysis_report)
Feb-18 22:36:33.654 [main] DEBUG nextflow.Session - Session await > all process finished
Feb-18 22:36:33.752 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Feb-18 22:36:34.612 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm)
Feb-18 22:36:34.612 [main] DEBUG nextflow.Session - Session await > all barriers passed