nextflow-io/nextflow

A DSL for data-driven computational pipelines

ardydavari

@ardydavari

I have a newbie question regarding passing the same variable multiple times to a command.
```

srr_ch = Channel.from( "SRR519926", "SRR1553607")

 process fastqdump {
   container 'quay.io/biocontainers/parallel-fastq-dump:0.6.5--py_0'

   input:
   each srr from srr_ch
   output:
   file("*.fastq") into fastq_ch

   """
   prefetch ${srr} && parallel-fastq-dump -t 8 -X 10000 --split-files -s ${srr}
   """
   }

Channel `srr_ch` has been used twice as an input by process `fastqdump` and another operator

ardydavari

@ardydavari

I believe i found the error:

srr_ch = Channel.from( "SRR519926", "SRR1553607")
srr_ch.println()


 process fastqdump {
   container 'quay.io/biocontainers/parallel-fastq-dump:0.6.5--py_0'

   input:
   each srr from srr_ch
   output:
   file("*.fastq") into fastq_ch

   """
   prefetch ${srr} && parallel-fastq-dump -t 8 -X 10000 --split-files -s ${srr}
   """
   }

println() was consuming the channel prior to routing to fastqdump

Paolo Di Tommaso

@pditommaso

indeed!

Ashley S Doane

@DoaneAS

Hi @pditommaso and all. I'm running a nextflow pipeline for realigning bam files. I have 4000+ samples, and ~2600 of these have completed. Running nextflow -resume main.nf crashed our interactive server, which has 192 CPUs and 10T RAM.

The admins told me that system-wide CPU use from nextflow processes was over %1000. I am using sge as executor and the crash seems to happen when nextflow is reading which samples are cached. Here's output when system became unstable:

executor >  sge (3)
[6e/65371e] process > speedyseq (EGAR00001156495_10698_5_7_cram) [100%] 2613 of 2616, cached: 2613

Rad Suchecki

@rsuchecki

Is that about excess cpu use on the head node? Can you submit your nextflow process as a job? @DoaneAS

Rad Suchecki

@rsuchecki

Paolo et al will correct me if I am wrong but I think one possibility could be that JVM heap space was running low, leading to excessive GC activity. Is pipeline code available to view? @DoaneAS

Daniel E Cook

@danielecook

Anyone know why Date.parse doesn't work?

Maxime Garcia

@MaxUlysse

Hi @pditommaso I have some follow up question on @drpatelh question here
I understand why we can't have traverse a directory over http, but is it possible to have something like https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file{1,2}.ext become two files: https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file1.ext https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file2.ext as it is with a regular path?

Paolo Di Tommaso

@pditommaso

yes, indeed

the glob is expanded to the actual files

Daniel E Cook

@danielecook

This works:

import java.text.SimpleDateFormat;
date_parse = new SimpleDateFormat("yyyy-MM-dd");
date_filter = date_parse.parse(params.date)

Paolo Di Tommaso

@pditommaso

well, datetime is a mess in any programing lang

Maxime Garcia

@MaxUlysse

It is?? I'll go check my tests then, I just tried and it wasn't working

Paolo Di Tommaso

@pditommaso

well, actually I'm not sure :joy:

Maxime Garcia

@MaxUlysse

It works

I must had another issue when I tried earlier

I'm guessing it'll work as well with s3://

Paolo Di Tommaso

@pditommaso

s3 can be traversed as a regular fs

Maxime Garcia

@MaxUlysse

thanks

Ashley S Doane

@DoaneAS

@rsuchecki thanks, yes pipeline is here: https://github.com/DoaneAS/realign.nf

Paolo Di Tommaso

@pditommaso

there could be something wrong in your script, NF is not supposed all that CPU

Daniel E Cook

@danielecook

@pditommaso agreed - but I'm curious why the groovy Date module doesn't work? Is it not imported?

Paolo Di Tommaso

@pditommaso

actually I'm not aware of it, therefore is not imported :smile:

is tehre a groovy-date module ?

Ashley S Doane

@DoaneAS

@rsuchecki pretty simple nextflow with only 1 process actually. Just takes a sampleindex.csv file with sample name, sample type (tumor or normal), and bam file path, and does a realignment with speedseq. The executor is sge, and the error seems to happen when nextflow is determining which results are cached.

Daniel E Cook

@danielecook

err maybe not - but the language spec suggests you should be able to do just Date.parse(format, input)

http://docs.groovy-lang.org/docs/groovy-2.4.4/html/groovy-jdk/java/util/Date.html#parse(java.lang.String,%20java.lang.String)

examples also present here: http://groovy-lang.org/single-page-documentation.html; Just curious maybe I'm missing something here

Paolo Di Tommaso

@pditommaso

can't check now, you may wont to report an issue

Daniel E Cook

@danielecook

I can do that

thanks

Paolo Di Tommaso

@pditommaso

welcome

Maxime Garcia

@MaxUlysse

OK, so I made a mistake earlier, it is not working in fact

I'll make a minimal example and an issue

Sri Harsha Meghadri

@harshameghadri

Hey folks, I am pretty new to using docker and singularity. I want to use the nf-core/rnaseq for my analysis. I consistently get errors while trying to execute this command singularity pull --name nf-core-rnaseq-1.3.img docker://nf-core/rnaseq:1.3

Unable to pull docker://nf-core/rnaseq:1.3: conveyor failed to get: Error reading manifest 1.3 in docker.io/nf-core/rnaseq: errors:

I am trying to pull to rackham. My analysis needs to be executed on bianca. Any tips are super appreciated.

Maxime Garcia

@MaxUlysse

try singularity pull --name nf-core-rnaseq-1.3.img docker://nfcore/rnaseq:1.3 instead, without the - in nf-core

I'm guessing if you have more question about nf-core pipelines, it should be better on our slack: https://nf-co.re/join

Sri Harsha Meghadri

@harshameghadri

I tried that as well getting the same error @MaxUlysse

Maxime Garcia

@MaxUlysse

You tried that on bianca?

Sri Harsha Meghadri

@harshameghadri

nope on rackham, I guess it needs internet

Maxime Garcia

@MaxUlysse

Sure

Just trying to find an easy mistake, sorry ;-)

Have you tried running that on an interactive node?

Sri Harsha Meghadri

@harshameghadri

hmmm on rackham? I dnt have allocation there.

Where communities thrive

People

Repo info

Activity