These are chat archives for nextflow-io/nextflow

1st
Jun 2018
Maxime HEBRARD
@mhebrard
Jun 01 2018 03:49
hello ! I am qute new with nextflow and I try to wrap my head around runing a nextflow script in a singularity image
my nextflow script need some input files .... --reads 'path/to/files' ... I didn't find a clear tuto to how I can bind my data folder into my singularity image
Maxime HEBRARD
@mhebrard
Jun 01 2018 04:00
running that on windows / vagrant / nextflow/ singularity ... I might be a bit lost in the multiple layer of VM .... in simple way : from where I execute the nextflow run, I have a /data folder with my files ... how can I mount this folder to ensure it is accessible inside singularity image ?
Radoslaw Suchecki
@bioinforad_twitter
Jun 01 2018 04:01
Hey @mhebrard, you don't
if nextflow can access the files
than so can the singularity container executed by nextflow
Maxime HEBRARD
@mhebrard
Jun 01 2018 04:03
hmm strange ... nextflow say "cannot find" .. is there a config to verify on singularity hand ?
Radoslaw Suchecki
@bioinforad_twitter
Jun 01 2018 04:04
does it work without singularity?
Maxime HEBRARD
@mhebrard
Jun 01 2018 04:05
well I do not have all the stack installed outside of singularity ... but yes that might be an issue with my path (not linked to singularity wrapping)
Radoslaw Suchecki
@bioinforad_twitter
Jun 01 2018 04:06
fine not to have all the tools locally, but I would try on a simple example to see if you can get to the files
Maxime HEBRARD
@mhebrard
Jun 01 2018 04:07
hmm yes I test my input files at first so don't need anything installed ... without singularity, it do not work either ^^"
thanks for the hint ... going back to work on my nextflow file ;)
Radoslaw Suchecki
@bioinforad_twitter
Jun 01 2018 04:09
:+1:
Maxime HEBRARD
@mhebrard
Jun 01 2018 05:36
@bioinforad_twitter that was a silly mistake in the path ... this problem is solved ;) (go to the next one lol)
Maxime HEBRARD
@mhebrard
Jun 01 2018 06:38
another newbee question ... I hace a channel define with Channel.fromFilePairs(params.reads).into{c_reads} ... if I print the channel, I see my list of files grouped by pair (as expected - 2 groups of 2 files for test) ... then I have one process that input file(reads) from c_reads ...
when I run nextflow, I see 1 submitted process... but I would expected 2 process to run in parallel ... one process for each group of input
any idea why I don't see my 2 process submitted in // ?
Maxime HEBRARD
@mhebrard
Jun 01 2018 06:50
note: my second process start after a while... looks like my process are called in series... is it a normal behavior for Channel.fromFilePairs ? then how can I submit my pairs in parallel ?
Paolo Di Tommaso
@pditommaso
Jun 01 2018 07:29
too little information, how is the process input composed? what executor are you using? config settings?
Maxime HEBRARD
@mhebrard
Jun 01 2018 07:50
Channel.fromFilePairs(params.reads).into{c_reads}
...
input: set val(name), file(reads) from c_reads
...
on windows in a Ubuntu VM managed with vagrant
nextflow run script.nf -with-singularity image.img --reads 'data/_R{1,2}'
...
files: SA_R1.fq, SA_R2.fq, SB_R1.fq, SB_R2.fq
...
no config file yet
Paolo Di Tommaso
@pditommaso
Jun 01 2018 07:50
Ubuntu VM => 1 cpu, I guess
Maxime HEBRARD
@mhebrard
Jun 01 2018 07:52
ohhh true ... I did config a VM for multi thread... but I forgot to config the vagrant VM ...
sorry for that >.>
Paolo Di Tommaso
@pditommaso
Jun 01 2018 07:52
No, pb. you are welcome
Maxime HEBRARD
@mhebrard
Jun 01 2018 08:09
yep... after config the VM to be multi-threads, my process are submitted in parallel :)
Paolo Di Tommaso
@pditommaso
Jun 01 2018 08:11
:clap:
Luca Cozzuto
@lucacozzuto
Jun 01 2018 08:33
Hi @pditommaso , here the error
N E X T F L O W  ~  version 0.29.1
Launching `rnaseq.nf` [admiring_fermat] - revision: 87620da4fc
ERROR ~ expecting '}', found ',' @ line 202, column 41.
    into read_length_for_trimming, read_len
                                 ^

1 error
Paolo Di Tommaso
@pditommaso
Jun 01 2018 08:34
now there's the error, but not the code => no answer :joy:
Luca Cozzuto
@lucacozzuto
Jun 01 2018 08:35
Santa pazienza
process getReadLength {
    input:
    file(single_read_pairs) from read_files_for_size.first()

    output:
    stdout into read_length_for_trimming, read_length_for_index, read_length_for_profile

    script:
    QC.getReadSize(single_read_pairs)
}
Paolo Di Tommaso
@pditommaso
Jun 01 2018 08:35
Holy patience :smile:
Luca Cozzuto
@lucacozzuto
Jun 01 2018 08:35
pig Maremma
Paolo Di Tommaso
@pditommaso
Jun 01 2018 08:36
I guess is complaining here
 output:
    stdout into read_length_for_trimming, read_length_for_index, read_length_for_profile
use
 output:
    stdout into (read_length_for_trimming, read_length_for_index, read_length_for_profile)
Luca Cozzuto
@lucacozzuto
Jun 01 2018 08:36
well
this I cannot understand
Paolo Di Tommaso
@pditommaso
Jun 01 2018 08:36
nextflow-io/nextflow#141
Luca Cozzuto
@lucacozzuto
Jun 01 2018 08:36
sometimes it work without sometimes with
ok
Maxime HEBRARD
@mhebrard
Jun 01 2018 09:01
Yeah my first workflow succeeded \o/ thanks nextflow ;)
Paolo Di Tommaso
@pditommaso
Jun 01 2018 09:01
:tada: :tada: :tada:
Bioninbo
@Bioninbo
Jun 01 2018 09:06
Hello Paolo and everyone,
When a process fails, there is an error message that makes it very easy to debug, with the path to the working directory of the process, and the failing commands. I was thinking that it could be even better to add the command used to launch the container in a shell, or at least the name of the container used, in case a container was used by the failing process. Otherwise it can be a bit more tedious and repetitive to find the right path to the container used. What do you think?
Luca Cozzuto
@lucacozzuto
Jun 01 2018 09:19
it is in .command.run
normally when a process fails you can re-run by launching .command.run (in a cluster just qsub .command.run)
Bioninbo
@Bioninbo
Jun 01 2018 09:25
weird I don't see it (the path to the container) in .command.run
Luca Cozzuto
@lucacozzuto
Jun 01 2018 09:26
uh
at the end you should have

tee2=$!
(
set +u; env - PATH="$PATH" SINGULARITYENV_TMP="$TMP" SINGULARITYENV_TMPDIR="$TMPDIR" singularity exec /software/bi/biocore_tools/gi
t/singularity_conversion/singularity/biocorecrg_rnaseq-2018-01-09-7fb3d092dd20.img /bin/bash -c "cd $PWD; /bin/bash -ue /nfs/softwa
re/bi/biocore_tools/git/nextflow/RNAseq/work/88/4665fab54e8f3ce3b626f9b3bee071/.command.sh"
) >$cout 2>$cerr &
pid=$!
wait $pid || ret=$?
wait $tee1 $tee2
with the path and everything
Bioninbo
@Bioninbo
Jun 01 2018 09:27
Ah right, I forgot to add the container for the process. -_- Ok thanks for info then!
Luca Cozzuto
@lucacozzuto
Jun 01 2018 09:27
:)
Maxime Garcia
@MaxUlysse
Jun 01 2018 10:03
@Bioninbo You can also specify whatever you want to see with the workflow.onErrorhttps://www.nextflow.io/docs/latest/metadata.html#error-handler
You can access the workflow.container that will contain all containers for your pipeline
That could be easier than looking at the .command.run file everytime
Maxime Garcia
@MaxUlysse
Jun 01 2018 10:08
You can have a look at all the different possible properties here: https://www.nextflow.io/docs/latest/metadata.html#runtime-metadata
Bioninbo
@Bioninbo
Jun 01 2018 10:09
I see, that's great. Thanks @MaxUlysse !
Maxime Garcia
@MaxUlysse
Jun 01 2018 10:09
:D
micans
@micans
Jun 01 2018 13:37
Hopefully I can phrase this question clearly. Yesterday a cram extraction resulted in a single fastq file (should have been two). I then made a tiny main.nf file where I simulated random (low frequency) failures. I inserted a filter, .filter { line, files -> files.size() == 2 } .set { fox_filtered }, and this works. However, when I resume this pipeline, the cases that failed (generated a single file instead of two) are not retried. So is it possible to filter out / ignore those cases in such a way that they will be retried? I guess it is a test I can put in the script section. Is that the best solution?
Paolo Di Tommaso
@pditommaso
Jun 01 2018 13:39
However, when I resume this pipeline, the cases that failed (generated a single file instead of two) are not retried
if are not retried, they have not failed ..
micans
@micans
Jun 01 2018 13:40
yes you are right. Asking the question here has helped to unjumble my brain.
Paolo Di Tommaso
@pditommaso
Jun 01 2018 13:41
Nf chat improves brain activity :smile:
micans
@micans
Jun 01 2018 13:46
Looking back -- the error I got had this in it: --readFilesIn <(gzip -dc 4101STDY6462102_1.fastq.gz) <(gzip -dc null). Now there is one stupid error I made (process substitution in a bsub command), but I traced the null to the cram file producing a single fastq file. So I should investigate why samtools thought it succeeded. (when I reran it manuall it proceeded two files ... typical). Thanks :-)
Let me know if this is too much information, but this was the situation:
bc-3|56/14b6ccab75a9ff2309bedf258c009d ls
4101STDY6462102_1.fastq.gz
bc-3|56/14b6ccab75a9ff2309bedf258c009d ls -a
.   4101STDY6462102_1.fastq.gz  .command.err  .command.out  .command.sh    .command.trace
..  .command.begin              .command.log  .command.run  .command.stub  .exitcode
bc-3|56/14b6ccab75a9ff2309bedf258c009d cat .command.sh
#!/bin/bash -euo pipefail
samtools merge -f 4101STDY6462102.cram 20485_8#3.cram
# check that the size of the cram file is >0.5Mb
minimumsize=500000
actualsize=$(wc -c <"4101STDY6462102.cram")
if [ $actualsize -ge $minimumsize ]; then
    samtools sort \
        -n \
        -@ 4 \
        -m 1110612736 \
        4101STDY6462102.cram | \
        samtools fastq \
            -N \
            -@ 4 \
            -1 4101STDY6462102_1.fastq.gz \
            -2 4101STDY6462102_2.fastq.gz \
            -
fi
bc-3|56/14b6ccab75a9ff2309bedf258c009d cat .exitcode
0bc-3|56/14b6ccab75a9ff2309bedf258c009d echo $(cat .exitcode)
0
Anyway not a NF situation!
Paolo Di Tommaso
@pditommaso
Jun 01 2018 13:51
Anyway not a NF situation!
that's reassuring :joy:
micans
@micans
Jun 01 2018 13:53
Like I said, give it a few months before I ask something demanding :smile: off to a lecture now, have a great weekend. Still learning about NF, fantastic tool.
Paolo Di Tommaso
@pditommaso
Jun 01 2018 13:53
thanks, same there
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:07

Hi,
I'm trying to make a function (several in fact), and I screwed up something somewhere, and I got this error which is not very helpfull:

ERROR ~ BUG! exception in phase 'semantic analysis' in source unit '_nf_script_bd1c0f9b' The lookup for QC caused a failed compilaton. There should not have been any compilation from this call.

QC being my library where all my functions are

Is there anyway to get a more detailed error message?
Paolo Di Tommaso
@pditommaso
Jun 01 2018 14:07
wow !
is there the error trace in the log file ?
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:08
Yes, good point
Found it
It's because I'm using task.memory.toGiga inside the code for my function
Paolo Di Tommaso
@pditommaso
Jun 01 2018 14:10
:+1:
how is the code ?
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:11
// Run bamQC on vcf file
  static def bamQC(bam,idSample) {
    """
    qualimap --java-mem-size=${task.memory.toGiga()}G \
    bamqc \
    -bam ${bam} \
    -outdir ${idSample} \
    -outformat HTML
    """
  }
Paolo Di Tommaso
@pditommaso
Jun 01 2018 14:12
ah, can't do that because task is not defined in that scope (tho, surely a strange error message)
however you can do the following
 static def bamQC(bam, idSample, mem) {
    """
    qualimap --java-mem-size=${mem.toGiga()}G \
    bamqc \
    -bam ${bam} \
    -outdir ${idSample} \
    -outformat HTML
    """
  }
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:13
smart
Paolo Di Tommaso
@pditommaso
Jun 01 2018 14:13
well, this is programming 1-0-1 :smile:
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:15
But I code similar error whenever I try some stuff with libraries, now I'll know to look at the .nextflow.log for a more detailed error
Thanks ;-)
Paolo Di Tommaso
@pditommaso
Jun 01 2018 14:15
always look there! ;)
Maxime Garcia
@MaxUlysse
Jun 01 2018 14:17
working, and I did use a similar setting when using a params inside a function as well, I should have thought of that
Luca Cozzuto
@lucacozzuto
Jun 01 2018 15:17
have a look at .nextflow.log
you have the explanation of what it fails
I'm doing the same :)
    static def qualimapRNAseq ( bamfile, annotation_file, outfolder="QUALIMAP", strand="strand-specific-reverse", memory="2G", read_single="NO", extrapars="") { 

    """
       pe_mode="";
       if [ `echo ${read_single} == "NO"` ]; then pe_mode="-pe"; fi

       unset DISPLAY
       mkdir tmp
       export JAVA_OPTS="-Djava.awt.headless=true -Xmx${memory} -Djava.io.tmpdir=\$PWD/tmp"    
       qualimap rnaseq \${pe_mode} --java-mem-size=${memory} -bam ${bamfile} -gtf ${annotation_file} -outdir ${outfolder} -p ${strand} ${extrapars}
       rm -fr tmp
    """