These are chat archives for nextflow-io/nextflow

21st
Feb 2018
Stephen Zhang
@zsteve
Feb 21 2018 01:41
Hi all, I'm trying to deal with an R script that only works with absolute paths. Within my shell: block I am trying to do the following:
echo $(pwd). However, nextflow is throwing me an error about the $ sign. Usage of echo \$(pwd) doesn't help. Any idea of how to perform this?
mgjmg
@mgjmg
Feb 21 2018 07:38
@vsoch hmm, thanks; tried snakemake and ... have turned to NF! (-: It's early days but it looks promising; if nf wasn't running something you expect, take a look at the .command.run file for the process; the docker run command NF uses is explicit there - you can work from that see what your container does from the commandline directly
Paolo Di Tommaso
@pditommaso
Feb 21 2018 08:09
@zsteve all of these works
process foo {
  shell:
  '''
  echo $(pwd)
  '''
}
process foo {
  shell:
  '''
  echo $PWD
  '''
}
process foo {
  shell:
  """
  echo \$PWD
  """
}
process foo {
  shell:
  """
  echo \$(pwd)
  """
}
Bioninbo
@Bioninbo
Feb 21 2018 10:21
Hello. I have an issue with the config file. When I put testvar = 3 in the nextflow.config file and println "testvar: ${testvar}" in the main.nf file I get the error No such variable: testvar . I tried importing the nextflow.config file with includeConfig but it didn't work. What am I doing wrong here?
Paolo Di Tommaso
@pditommaso
Feb 21 2018 10:22
variables in the config are not exported in the pipeline script
only params.xxx are
Bioninbo
@Bioninbo
Feb 21 2018 10:23
Ah I see, thanks!
Paolo Di Tommaso
@pditommaso
Feb 21 2018 10:23
:+1:
Mamana
@mypandos
Feb 21 2018 13:08
@pditommaso I am experiencing something strange using different version of NF, looks like my cache from NF 0.27.1 is being ignored when I used NF 0.27.6. Is this a normal behaviour?
Maxime Borry
@maxibor
Feb 21 2018 13:37
Hello,
I have a process that emits paired_end files (trimming of PE fastq), and I'd like to get them in my next process (aligning), can I use fromFilePairs again ?
Here how it would look like in my mind (inspired by the NF documentation):
params.reads = "/data/*_{1,2}.fastq.gz"

Channel
    .fromFilePairs( params.reads, size: 2 )
    .into {raw_reads}

process trimming{

    input:
        set val(name), file(reads) from raw_reads
    output:
        set val(name), file("*.fastq") into trimmed_reads
    script:
    // creates $name.trimmed.r1.fastq and $name.trimmed.r2.fastq
        """
        mytrimmer -r1 ${reads[0]} -r2 ${reads[1]} -o $name
        """
}

process aligner {
    input:
        set val(name), file(trimmed) fromFilePairs(trimmed_reads, size: 2) { file -> "*.trimmed.r{1,2}.fastq" }
    output:
        set val(name), file("*.sam") into aligned
    script:
    // creates $name.sam
        """
        myaligner -r1 ${trimmed[0]} -r2 ${trimmed[1]} -o $name
        """
}
Evan Floden
@evanfloden
Feb 21 2018 13:48
Sorry, I may has misread. You could try:
process aligner {
    input:
        set val(name), file(trimmed_read1), file(trimmed_read2) from trimmed_reads
if you have the process trimming as follows:
process trimming{

    input:
        set val(name), file(reads) from raw_reads
    output:
        set val(name), file("$name.trimmed.r1.fastq”), file(“$name.trimmed.r2.fastq") into trimmed_reads
    script:
    // creates $name.trimmed.r1.fastq and $name.trimmed.r2.fastq
        """
        mytrimmer -r1 ${reads[0]} -r2 ${reads[1]} -o $name
        """
}
Maxime Borry
@maxibor
Feb 21 2018 13:53
Yes, they are the output of process trimming
Thanks a lot @skptic , I'll try that !
Maxime Borry
@maxibor
Feb 21 2018 14:39
Hum, there seems to be an issue with this solution when I test with real data and code:
I get an ERROR ~ No such variable: r1
params.reads = "/home/maxime/Desktop/data/test.r{1,2}.fastq.gz"

CPU = 8

Channel
    .fromFilePairs( params.reads )
    .ifEmpty { error "Cannot find any reads matching: ${params.pairs}" }
    .set { raw_reads }

process trimming{
    tag "$name"
    cpus = CPU

    input:
        set name, file(reads) from raw_reads
    output:
        set val(name), file("$name.r1.trimmed.fastq"), file("$name.r2.trimmed.fastq") into trimmed_reads
    script:
        out1 = name+".r1.trimmed.fastq"
        out2 = name+".r2.trimmed.fastq"
        """
        AdapterRemoval --basename $name --file1 ${reads[0]} --file2 ${reads[1]} --output1 $out1 --output2 $out2 --threads ${task.cpus}
        """
}

process aligner {
    tag "$name"
    cpus = CPU
    input:
        set val(name), file(trim_r1), file(trim_r2) from trimmed_reads
    output:
        set val(name), file("*.sam") into aligned
    script:
    // creates $name.sam
        """
        bowtie2 -x  /home/maxime/Documents/db/organellome/bowtie2index/organellome -r1 $trim_r1 -r2 $trim_r2 -S $name -p ${task.cpus}
        """
}
(organdiet) maxime@gph:~/Desktop$ nextflow run test.nf 
N E X T F L O W  ~  version 0.27.4
Launching `test.nf` [disturbed_rosalind] - revision: f5380892b7
[warm up] executor > local
ERROR ~ No such variable: r1

 -- Check script 'test.nf' at line: 17 or see '.nextflow.log' file for more details
it seems that the $ character somehow escapes the string type for the whole string...
jncvee
@jncvee
Feb 21 2018 14:42
I want to get a specific file from a channel.fromPath() by putting the specific input file name in the input but it is not working
Maxime Borry
@maxibor
Feb 21 2018 14:43
Can you provide an example @jncvee ?
(here is one from the Nextflow examples https://github.com/nextflow-io/examples/blob/master/blast.nf )
Maxime Borry
@maxibor
Feb 21 2018 14:49
if the channel is the output of a process, maybe collectFile can do the job ? https://github.com/nextflow-io/examples/blob/master/split_and_collect.nf
jncvee
@jncvee
Feb 21 2018 15:23
I'm using a channel that has a path for all .txt files in a folder. And I want the input to be a specific .txt file from the folder. however, the process keeps using all the .txt files
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:24
what the problem?
Bioninbo
@Bioninbo
Feb 21 2018 15:25
Hello. I runned my script with one container per process and got many crashes since I am in development phase. However I end up with multiple instances of mounted containers that takes all the memory. Is this a normal behavior in development?
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:26
is not normal in the extend is crash ..
try reducing the number of parallel task using -qs 2
Bioninbo
@Bioninbo
Feb 21 2018 15:27
I set qs to 1 already
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:27
so it won't run more than a task at time
Bioninbo
@Bioninbo
Feb 21 2018 15:30
Each time I run the script (the workflow fail) I end up with two more mounted volumes
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:30
are you using docker ?
Bioninbo
@Bioninbo
Feb 21 2018 15:31
No Singularity
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:31
what is the error message ?
Bioninbo
@Bioninbo
Feb 21 2018 15:33
[bam_header_read] EOF marker is absent. The input is probably truncated.
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:33
does not help much
Bioninbo
@Bioninbo
Feb 21 2018 15:33
I use a samtools container
It probably do not unmount after the crash I guess
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:34
can help here, my suggestion is to change in the task work dir and troubleshoot the problem running bash .command.run
Bioninbo
@Bioninbo
Feb 21 2018 15:34
Ok I'll try that
thanks for your help!
Paolo Di Tommaso
@pditommaso
Feb 21 2018 15:34
:+1:
Evan Floden
@evanfloden
Feb 21 2018 15:36
@maxibor I think you may have to specify the variable and string like: file(“${name}.r1.trimmed.fastq”)
Phil Ewels
@ewels
Feb 21 2018 15:59
@pditommaso - quick opinion question before I start playing: I think it should be possible to use workflow.scriptName in nextflow.config to pull in different config files using different includeConfig statements depending on which nextflow script within a repository is being run
any running order problems or anything which would prevent this from working that you can think of?
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:01
not sure, workflow metadata was not designed to create dynamic configurations
you are in uncharted waters
I think it's not available at all
Phil Ewels
@ewels
Feb 21 2018 16:04
yeah, that's what I was worried about
I have a pipeline with two scripts, and now I get tonnes of warning log messages about processes being defined in the config but not present in the pipeline
doesn't prevent the pipeline from working, but it's annoying :P
eg. WARN: The config file defines settings for an unknown process:
so was wondering if there was a way to split these configs into two files and load them conditionally depending on which script is being run
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:06
maybe using multiple profiles could a better approach
upcoming version allows multiple profiles
nextflow-io/nextflow#370
Maxime Garcia
@MaxUlysse
Feb 21 2018 16:07
That will be interesting
Phil Ewels
@ewels
Feb 21 2018 16:07
aha, yes that would be perfect :+1:
hah, except it would be good to have a default one for both "sets"
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:07
try the 0.28.0-SNAPSHOT version
that should be possible
Maxime Garcia
@MaxUlysse
Feb 21 2018 16:08
@ewels I have the same problem since dissecting Sarek into several scripts
Phil Ewels
@ewels
Feb 21 2018 16:09
@MaxUlysse - yeah exactly, that was part of what I was thinking

that should be possible

possible already or in a future version?

use case here is to have a profile for aligner 1 and a profile for aligner 2. Then additionally have different profiles for different compute settings.
So would be good to have a defatult (standard) profile for both aligner type and also compute setting.
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:10
a couple of weeks, users feedback can improve confidence and speedup the process ;)
Phil Ewels
@ewels
Feb 21 2018 16:11
hah, ok cool :+1:
In fairness, we currently have a -profile none which doesn't import anything
so the default would be annoying for that case
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:12
why ?
Phil Ewels
@ewels
Feb 21 2018 16:12
so maybe I can just throw a big warning if nothing is configured (and even exit)
I added that one so that people can set up their own configs in their home directories
and not have to worry about overwriting everything
but I'm not sure if anyone uses it..
jncvee
@jncvee
Feb 21 2018 16:34
what are the specific commands to make nextflow slurm compatible
Paolo Di Tommaso
@pditommaso
Feb 21 2018 16:36
they are already compatible
jncvee
@jncvee
Feb 21 2018 16:39
do I still need to enable it ? because the documentation says I need to do that. And if so how do i specifically do that
Kevin Sayers
@KevinSayers
Feb 21 2018 16:46
@jncvee yes you need to enable it, add process.executor = 'slurm' in your nextflow.config file
Chris Fields
@cjfields
Feb 21 2018 17:05
I have a general best practices question. We have a workflow with two forks using the same tool. This generates a generic file name that collides downstream (the tool is Circlator and the filename is, horribly, '06.fixstart.fasta'). So obvious solution would be to rename uniquely based on the process and tool, then send it in a Channel for downstream.
Are users renaming via the job script (in bash for example) or using the file.renameTo()? https://www.nextflow.io/docs/latest/script.html?highlight=rename#rename-files
I could easily rename, just wasn't sure whether someone had a Nextflow-y way of doing this in the output block of the process.
Phil Ewels
@ewels
Feb 21 2018 17:15
So just put a * in the input filename specification and nextflow will automatically insert a number to ensure that the filenames don't clash
@pditommaso: Just tried NXF_VER=0.28.0-SNAPSHOT nextflow run and still got Can only specify option -profile once. -- Check the available commands and options and syntax with 'help'
aha, maybe the installation failed?
CAPSULE: Transfer failed: capsule.org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact io.nextflow:nxf-httpfs:pom:0.28.0-20180219.081654-18 in local (file:/Users/philewels/.m2/repository) (for stack trace, run with -Dcapsule.log=verbose)
yes:
$ NXF_VER=0.28.0-SNAPSHOT nextflow -version

      N E X T F L O W
      version 0.27.1 build 4755
      last modified 18-01-2018 12:43 UTC (13:43 CEST)
      cite doi:10.1038/nbt.3820
      http://nextflow.io