nextflow-io/nextflow

A DSL for data-driven computational pipelines

Stijn van Dongen

@micans

You could do nextflow run pipe/repo --opsFile opsFile.in and inside the NF script prepend workflow.projectDir (or the other one), or have that controlled by yet another parameter.

mmatthews06

@mmatthews06

Good point. That may be a good common practice for referencing files that usually live in the repo itself. (Granted this may not need to be a parameter at all, I'm just trying things out.)

so this worked:

export NXF_HOME=~/.nextflow
nextflow run pipe/repo --opsFile $NXF_HOME/assets/pipe/repo/opsFile.in

Stijn van Dongen

@micans

In that case you could have something more semantic, e.g. --ops MODE and internally map that in NF to opsMODE.in, i.e. loosen the strict file name specification.

mmatthews06

@mmatthews06

You could do nextflow run pipe/repo --opsFile opsFile.in and inside the NF script prepend workflow.projectDir (or the other one), or have that controlled by yet another parameter.

I think I like this idea better.

Good suggestions. I appreciate the help.

Stijn van Dongen

@micans

Cool, I always learn something from discussions here :+1:

mmatthews06

@mmatthews06

Best solution here: since the file in the repo is the default, in the pipeline, simply set params.opsFile = "$workflow.projectDir/opsFile.in". Theoretically all non-default files would come from outside the repo. It was set to just params.opsFile = "opsFile.in", which works when you're developing inside the repository directory, but not once you check it out. Seems obvious, now.

HashRocketSyntax

@HashRocketSyntax

Or, when handling spot instance interrupt, should a resume be attempted on another server (is that possible?)

mmatthews06

@mmatthews06

On a completely different note, version 19.07.0 is giving me GitHub authorization issues that version 19.04.1 is not. Or, sometimes, this error:

WARN: Cannot read project manifest -- Cause: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

mmatthews06

@mmatthews06

^ Never mind, I'll try this: https://www.nextflow.io/docs/latest/sharing.html#private-server-configuration

mmatthews06

@mmatthews06

^ Ok, these instructions seem to have worked quite well: https://jfrog.com/knowledge-base/how-to-resolve-unable-to-find-valid-certification-path-to-requested-target-error/

ardydavari

@ardydavari

I have a newbie question regarding passing the same variable multiple times to a command.
```

srr_ch = Channel.from( "SRR519926", "SRR1553607")

 process fastqdump {
   container 'quay.io/biocontainers/parallel-fastq-dump:0.6.5--py_0'

   input:
   each srr from srr_ch
   output:
   file("*.fastq") into fastq_ch

   """
   prefetch ${srr} && parallel-fastq-dump -t 8 -X 10000 --split-files -s ${srr}
   """
   }

Channel `srr_ch` has been used twice as an input by process `fastqdump` and another operator

ardydavari

@ardydavari

I believe i found the error:

srr_ch = Channel.from( "SRR519926", "SRR1553607")
srr_ch.println()


 process fastqdump {
   container 'quay.io/biocontainers/parallel-fastq-dump:0.6.5--py_0'

   input:
   each srr from srr_ch
   output:
   file("*.fastq") into fastq_ch

   """
   prefetch ${srr} && parallel-fastq-dump -t 8 -X 10000 --split-files -s ${srr}
   """
   }

println() was consuming the channel prior to routing to fastqdump

Paolo Di Tommaso

@pditommaso

indeed!

Ashley S Doane

@DoaneAS

Hi @pditommaso and all. I'm running a nextflow pipeline for realigning bam files. I have 4000+ samples, and ~2600 of these have completed. Running nextflow -resume main.nf crashed our interactive server, which has 192 CPUs and 10T RAM.

The admins told me that system-wide CPU use from nextflow processes was over %1000. I am using sge as executor and the crash seems to happen when nextflow is reading which samples are cached. Here's output when system became unstable:

executor >  sge (3)
[6e/65371e] process > speedyseq (EGAR00001156495_10698_5_7_cram) [100%] 2613 of 2616, cached: 2613

Rad Suchecki

@rsuchecki

Is that about excess cpu use on the head node? Can you submit your nextflow process as a job? @DoaneAS

Rad Suchecki

@rsuchecki

Paolo et al will correct me if I am wrong but I think one possibility could be that JVM heap space was running low, leading to excessive GC activity. Is pipeline code available to view? @DoaneAS

Daniel E Cook

@danielecook

Anyone know why Date.parse doesn't work?

Maxime Garcia

@MaxUlysse

Hi @pditommaso I have some follow up question on @drpatelh question here
I understand why we can't have traverse a directory over http, but is it possible to have something like https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file{1,2}.ext become two files: https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file1.ext https://raw.githubusercontent.com/maxulysse/test-datasets/sarek/file2.ext as it is with a regular path?

Paolo Di Tommaso

@pditommaso

yes, indeed

the glob is expanded to the actual files

Daniel E Cook

@danielecook

This works:

import java.text.SimpleDateFormat;
date_parse = new SimpleDateFormat("yyyy-MM-dd");
date_filter = date_parse.parse(params.date)

Paolo Di Tommaso

@pditommaso

well, datetime is a mess in any programing lang

Maxime Garcia

@MaxUlysse

It is?? I'll go check my tests then, I just tried and it wasn't working

Paolo Di Tommaso

@pditommaso

well, actually I'm not sure :joy:

Maxime Garcia

@MaxUlysse

It works

I must had another issue when I tried earlier

I'm guessing it'll work as well with s3://

Paolo Di Tommaso

@pditommaso

s3 can be traversed as a regular fs

Maxime Garcia

@MaxUlysse

thanks

Ashley S Doane

@DoaneAS

@rsuchecki thanks, yes pipeline is here: https://github.com/DoaneAS/realign.nf

Paolo Di Tommaso

@pditommaso

there could be something wrong in your script, NF is not supposed all that CPU

Daniel E Cook

@danielecook

@pditommaso agreed - but I'm curious why the groovy Date module doesn't work? Is it not imported?

Paolo Di Tommaso

@pditommaso

actually I'm not aware of it, therefore is not imported :smile:

is tehre a groovy-date module ?

Ashley S Doane

@DoaneAS

@rsuchecki pretty simple nextflow with only 1 process actually. Just takes a sampleindex.csv file with sample name, sample type (tumor or normal), and bam file path, and does a realignment with speedseq. The executor is sge, and the error seems to happen when nextflow is determining which results are cached.

Daniel E Cook

@danielecook

err maybe not - but the language spec suggests you should be able to do just Date.parse(format, input)

http://docs.groovy-lang.org/docs/groovy-2.4.4/html/groovy-jdk/java/util/Date.html#parse(java.lang.String,%20java.lang.String)

examples also present here: http://groovy-lang.org/single-page-documentation.html; Just curious maybe I'm missing something here

Paolo Di Tommaso

@pditommaso

can't check now, you may wont to report an issue

Daniel E Cook

@danielecook

I can do that

thanks

Paolo Di Tommaso

@pditommaso

welcome

Maxime Garcia

@MaxUlysse

OK, so I made a mistake earlier, it is not working in fact

I'll make a minimal example and an issue

Where communities thrive

People

Repo info

Activity