These are chat archives for nextflow-io/nextflow

22nd
Feb 2019
Paolo Di Tommaso
@pditommaso
Feb 22 09:39
have a look to this post
or pass here with your laptop to configure it :)
Luca Cozzuto
@lucacozzuto
Feb 22 09:42
Hey Ernesto! Nice to see you again!
Qi ZHAO
@likelet
Feb 22 10:38

hi, a simple question. Why the $workDir in config file can not be recognized by NF

N E X T F L O W  ~  version 19.01.0
Launching `RNAseqPipe/main.nf` [thirsty_hamilton] - revision: a2f41a936f
This workflow requires Nextflow version 0.26 or greater -- You are running version 19.01.0
ERROR ~ GTF file not found: $workDir/Genome/gencode.chr22.gtf

 -- Check '.nextflow.log' file for details

image.png

micans
@micans
Feb 22 10:39
@likelet think it is workflow.workDir
"${workflow.workDir}"
Qi ZHAO
@likelet
Feb 22 10:42
thanks @micans seems not work either. i used it in the config file, am I wrong with this
N E X T F L O W  ~  version 19.01.0
Launching `RNAseqPipe/main.nf` [disturbed_aryabhata] - revision: a2f41a936f
ERROR ~ Unknown config attribute `params.workflow.workDir` -- check config file: /home/zhaoqi/RNAseqPipeTestData/RNAseqPipe/nextflow.config
and here is my config info
micans
@micans
Feb 22 10:44
well that says workflow is part of the params object ...
Qi ZHAO
@likelet
Feb 22 10:44
params{
    dapath = '/home/zhaoqi/software/DAtools.jar'
    star_index = "${workflow.workDir}/Genome/STARindex/hg38_chr22"
    gtf = "${workflow.workDir}/Genome/gencode.chr22.gtf"

    // GSEA setting
    gseapath="/home/Public/Database/human/GSEA_base/gsea/gsea-3.0.jar"
    gsea_pathway="/home/Public/Database/human/GSEA_base/gsea/GSEA_c2.all.v6.1.symbols.gmt"

    //test data
    read= "Fastq/*_{1,2}.fastq.gz"
    designfile="design.file"
    comparefile="compare.txt"
}
micans
@micans
Feb 22 10:44
I'll try to replicate ...
Qi ZHAO
@likelet
Feb 22 10:45
and also a question, how can I get the workDir from a default configuration in config file
micans
@micans
Feb 22 10:46
gtf = "${workflow.workDir}/Genome/gencode.chr22.gtf" looks a bit strange though ...
workDir is determined at runtime (possibly by -w option) ... either you have the gtf file in a fixed filesystem location, or you pull it in via a channel; why do you do it this way?
Paolo Di Tommaso
@pditommaso
Feb 22 10:47
*workflow* should not be used in the config
use workDir
micans
@micans
Feb 22 10:48
oh, sorry :-)
Qi ZHAO
@likelet
Feb 22 10:49
i used to set gtf = "Genome/gencode.chr22.gtf", it worked before. But it returns error (not found the file ) today when i rerun my pipe. So I try setting the wordDir to pass the current dir for analysis, and it failed
it's weird...
micans
@micans
Feb 22 10:50
that is a relative location; it depends on where you run it. Try setting an absolute fully qualified path name
I have (following nf-core rnaseq) gtf = "${params.genomes_base}/ensembl/GRCh38/Homo_sapiens.GRCh38.91.gtf"
Qi ZHAO
@likelet
Feb 22 10:51
absolute path works, but i want to make it adjust for other users when install test data
yes @micans , it need you setting the genomes_base in the config file as well
micans
@micans
Feb 22 10:52
Ah, I see what you mean
There is also projectDir, but I have it in the workflow object.
Qi ZHAO
@likelet
Feb 22 10:54
and it is particularly not applicable for those have different projectDir and workDir
micans
@micans
Feb 22 10:56
Would projectDir not be what you need? I assume that is where you ship the gtf file?
Qi ZHAO
@likelet
Feb 22 11:01
nope, it failed again.
anyway, thanks @micans i will see if there is another solution
micans
@micans
Feb 22 11:02
with anything todo with files I make a tiny example and play with that; good luck!
Qi ZHAO
@likelet
Feb 22 11:08
I think NF is okay with that (without workDir) , the current path could be passed to the file() function . Seems the error comes from the command inside the process. @micans thanks, no more question .
micans
@micans
Feb 22 11:08
Well, I listened, not much help. Always good to learn more myself!
Jonathan Manning
@pinin4fjords
Feb 22 12:18

Hi, I'm trying to output file names with numbers in them (reflecting different parameterisations), and I'm getting errors like: "Nul character not allowed: tsne_50.tsv". e.g. a process like:

process tsne_to_tsv {

    input:
        set val(perplexity), file(embeddings) from EMBEDDINGS_BY_PERPLEXITY

    output:
        set val(perplexity), file ("tsne_${perplexity}.tsv") into TSV_EMBEDDINGS                

    """
    cat $embeddings | sed 's/,/\t/g' > tsne_${perplexity}.tsv
    """
}

EMBEDDINGS_BY_PERPLEXITY contains tuples with value - filename pairs.

Are number-containing output files really not allowed? Using Nextflow 19.01.0.5050

Paolo Di Tommaso
@pditommaso
Feb 22 12:22
there's something odd, provide the complete error message are it's printed
Jonathan Manning
@pinin4fjords
Feb 22 12:47

@pditommaso So, error is like:

ERROR ~ Error executing process > 'tsne_to_tsv (1)'

Caused by:
  Nul character not allowed: tnse_25.tsv


 -- Check '.nextflow.log' file for details
WARN: Killing pending tasks (9)

The tuples are like:

[45, <path>/work/f6/a09e4f89effbf7b1056555af6870d6/embeddings_45.csv]
[40, <path>/work/97/948b0dbc0076a48c2b78d9f8843263/embeddings_40.csv]

Works fine if I output to e.g. 'tsne.tsv' rather than tsne_${perplexity}.tsv

Paolo Di Tommaso
@pditommaso
Feb 22 12:48
how is the the error stack trace in the log file ?
Jonathan Manning
@pinin4fjords
Feb 22 12:49
Feb-22 12:41:50.904 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'tsne_to_tsv (1)'

Caused by:
  Nul character not allowed: tnse_25.tsv

java.nio.file.InvalidPathException: Nul character not allowed: tnse_25.tsv
    at java.base/sun.nio.fs.UnixPath.checkNotNul(UnixPath.java:91)
    at java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:81)
    at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
    at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:280)
    at java.base/java.nio.file.Path.resolve(Path.java:402)
    at java.base/jdk.internal.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:74)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at nextflow.processor.TaskProcessor.collectOutFiles(TaskProcessor.groovy:1373)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:167)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:192)
    at nextflow.processor.TaskProcessor.collectOutputs(TaskProcessor.groovy:1314)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:192)
    at nextflow.processor.TaskProcessor.collectOutputs(TaskProcessor.groovy:1295)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
    at nextflow.processor.TaskProcessor.finalizeTask(TaskProcessor.groovy:1992)
    at n
... sorry @pditommaso - here's the end of the trace:
    at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:619)
    at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:539)
    at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:418)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1011)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:994)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
    at nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:302)
    at nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
    at groovy.lang.Closure.run(Closure.java:492)
    at java.base/java.lang.Thread.run(Thread.java:844)
Feb-22 12:41:50.918 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Nul character not allowed: tnse_25.tsv
Paolo Di Tommaso
@pditommaso
Feb 22 12:53
I'm perplexed ..
not sure to understand why it's happening, you may want to report a bug with a test case
however I think that a possibile workaround could be
    output:
        set val(perplexity), file ("tsne_*.tsv") into TSV_EMBEDDINGS
Jonathan Manning
@pinin4fjords
Feb 22 12:56
Okay, thanks @pditommaso , I'll try that. I'll also try to make a minimal reproducible example.
Tim Dudgeon
@tdudgeon
Feb 22 12:57

Hi, if I start a workflow like this nextflow run -process.executor ignite ... it starts fine.
But if I omit that option and instead add something like this to my nextflow.config file:

process {
    executor = 'ignite'
}

I get this error:
ERROR ~ Unknown executor name: ignite
Am I doing something stupid here?

Paolo Di Tommaso
@pditommaso
Feb 22 13:00
this is a quick thing, however only in the config is not enough for ignite
put NXF_EXECUTOR=ignite in the env
Tim Dudgeon
@tdudgeon
Feb 22 13:11
OK, that worked. Thanks.
So is this a bug or are the docs wrong?
Oren Tzfadia
@ortzf
Feb 22 13:13
Hello! I am building a NextFlow script aimed to be an 'automation glue script' which allow users running MOFA and WGCNA pipelines in tandem. See git: https://github.com/vibbits/NextFlow_pipelines
My question is - I would like to build this in a nicer way where the R scripts are not 'pasted' within the nf acript. Also instead of calling the installation commands for the required R packages, I wish to use a docker container (which we created for these pipelines, see: https://github.com/vibbits/rocker_conda_data_integration)
@pditommaso any suggestions?
Alexander Peltzer
@apeltzer
Feb 22 13:20
You can have the r scripts as they are in the bin folder of your pipeline and run them from inside the actual nextflow script
Rad Suchecki
@rsuchecki
Feb 22 13:20
@ortzf put your rscripts under bin/
Alexander Peltzer
@apeltzer
Feb 22 13:21
We add these R dependencies using bioconda recipes for example https://github.com/nf-core/rnaseq/blob/master/environment.yml and then use this environmen file to produce a Docker container
Rad Suchecki
@rsuchecki
Feb 22 13:22
This message was deleted
Jonathan Manning
@pinin4fjords
Feb 22 13:26

@ortzf We do what you're talking about by packaging scripts as Bioconda packages depending on the underlying R packages, e.g. https://github.com/ebi-gene-expression-group/bioconductor-scater-scripts. The bonus is that you get a container created automatically on quay.io.

e.g. the Bioconda recipe that pulls the above package is: https://github.com/bioconda/bioconda-recipes/tree/master/recipes/bioconductor-scater-scripts

Alexander Peltzer
@apeltzer
Feb 22 13:26
Exactly :+1:
Rad Suchecki
@rsuchecki
Feb 22 13:29
and if you'd rather avoid conda, here is an alternative example https://hub.docker.com/r/rsuchecki/rscripts/dockerfile
Jonathan Manning
@pinin4fjords
Feb 22 13:30
@pditommaso - bug report as requested: nextflow-io/nextflow#1044
Tim Dudgeon
@tdudgeon
Feb 22 13:51
Actually I'm wrong - defining env { NXF_EXECUTOR = 'ignite' } didn't work. It's using local executor :-(
Tim Dudgeon
@tdudgeon
Feb 22 13:58
If instead I define NXF_EXECUTOR as an environment variable in the shell where nextflow is run from I again get the ERROR ~ Unknown executor name: ignite error.
It only seems to work when added as a CLI parameter.
Paolo Di Tommaso
@pditommaso
Feb 22 13:59
set also NXF_MODE=ignite
Tim Dudgeon
@tdudgeon
Feb 22 14:16
OK, setting NXF_MODE=ignite as well works.
Paolo Di Tommaso
@pditommaso
Feb 22 14:28
:v:
Toni Hermoso Pulido
@toniher
Feb 22 14:32
@pditommaso Thanks! Blogpost helped a lot but I didn't manage yet. If I cannot solve it I will visit you maybe next week :P It looks related to awscli and custom AMI according to this: broadinstitute/cromwell#4604
Paolo Di Tommaso
@pditommaso
Feb 22 14:34
cromwell is surely wrong here :joy:
Toni Hermoso Pulido
@toniher
Feb 22 14:34
hehehe
The error is the same I get using Nextflow
Paolo Di Tommaso
@pditommaso
Feb 22 14:35
that's common docker error when it does not find a tool on the PATH
what's the container ?
it must be a container made by @lucacozzuto :joy:
Toni Hermoso Pulido
@toniher
Feb 22 14:44
well, one it's made by @lucacozzuto , but the other one is this https://www.nextflow.io/blog/2017/scaling-with-aws-batch.html :P
I will try others, just in case...
Luca Cozzuto
@lucacozzuto
Feb 22 14:50
@pditommaso my containers are the best. :P
Paolo Di Tommaso
@pditommaso
Feb 22 14:52
I know :D
Toni Hermoso Pulido
@toniher
Feb 22 15:29
@pditommaso well, it seems you are right. I tried these two ones https://github.com/nextflow-io/rnaseq-nf/ https://github.com/nextflow-io/blast-example and they seem to work. But https://github.com/nextflow-io/rnaseq-encode-nf and https://github.com/biocorecrg/vectorQC seem not with that Docker container error...
Luca Cozzuto
@lucacozzuto
Feb 22 15:49
so @toniher's fault! :PPPP
Toni Hermoso Pulido
@toniher
Feb 22 15:51
@lucacozzuto :( I was not using miniconda awscli
... in the ones that were failing. Everything under control now. Thanks @pditommaso for the hlep
Paolo Di Tommaso
@pditommaso
Feb 22 15:56
:clap: :clap:
bahnk
@bahnk
Feb 22 16:48
hello
Paolo Di Tommaso
@pditommaso
Feb 22 16:49
hello
bahnk
@bahnk
Feb 22 16:49
this is my first post here, so i would to thank Paolo for the fantastic job he's doing
nextflow is an amazingly helpful tool
Paolo Di Tommaso
@pditommaso
Feb 22 16:49
there's always a first time :D
thanks
bahnk
@bahnk
Feb 22 16:50
that being said i have a question :)
i used the -lib option to pass custom classes to nextflow
it's working very well
Paolo Di Tommaso
@pditommaso
Feb 22 16:50
yes
bahnk
@bahnk
Feb 22 16:50
however, i'm unable to resume
the taskprocessor is unable to find the custom class
Paolo Di Tommaso
@pditommaso
Feb 22 16:51
?
bahnk
@bahnk
Feb 22 16:51
is there a way to circumvent this? it sounds complicated though...
Paolo Di Tommaso
@pditommaso
Feb 22 16:52
ahh, it depends how you are using it
bahnk
@bahnk
Feb 22 16:52
sorry i forgot to mention something
i am creating a channel containing my custom classes
and then pass it to a process
Paolo Di Tommaso
@pditommaso
Feb 22 16:53
in that case the object must be serialisable
bahnk
@bahnk
Feb 22 16:54
ah interesting
you know what i'll write small example first
and then come back here
(thank you)
Paolo Di Tommaso
@pditommaso
Feb 22 16:55
welcome
bahnk
@bahnk
Feb 22 17:10
hi it's me again
package mypackage

class Fastq implements Serializable {

    String path
    String label

    Fastq(String path, String label) {
        this.path = path
        this.label = label
    }

    String getPath() {
        return this.path
    }

    String getLabel() {
        return this.label
    }
}
#!/usr/bin/env nextflow

import mypackage.Fastq

fastq1 = new Fastq("/path/one", "one")
fastq2 = new Fastq("/path/two", "two")

c = Channel.from(fastq1, fastq2)

process test {

    publishDir "output"

    input:
        val f from c

    output:
        file "*.txt" into files

    shell:
        """
        echo ${f.getPath()} > ${f.getLabel()}.txt
        """
}
CC = groovyc
ARCH = jar
OPT = cvf
SRC = src
PACKAGE = mypackage

.PHONY: .FORCE
.FORCE:

all: jar

jar: .FORCE $(PACKAGE)/Fastq.class
    $(ARCH) $(OPT) $(PACKAGE).jar $(PACKAGE)
    rm -v $(PACKAGE)/*

$(PACKAGE)/Fastq.class: .FORCE
    $(CC) -d . $(SRC)/Fastq.gy
that's the idea
i don't have any warning in this example though
bahnk
@bahnk
Feb 22 17:16
but with my full code i have :
Unable to resume cached task -- See log file for details
com.esotericsoftware.kryo.KryoException: Unable to find class: qcpipeline.Fastq
Stephen Kelly
@stevekm
Feb 22 18:21

@likelet

and also a question, how can I get the workDir from a default configuration in config file

I have been doing it like this at execution time:

    logdir="$(LOGDIR)/$(TIMESTAMP)" ; \
    export NXF_WORK="$${logdir}/work" ; \
    ./nextflow -log "$${logfile}" run main.nf
Stephen Kelly
@stevekm
Feb 22 18:57
I have seen that you can use nextflow -trace nextflow.executor run to get more detailed logs of the executor; what other things can you get logs from like this?
evanbiederstedt
@evanbiederstedt
Feb 22 19:23
I'm writing some unit tests for a PR
If I could ask the NF community a basic question, when writing tests with Spock...how do I see precisely what the tests were expecting?
I suspect this is largely ignorance on my part with respect to how to best develop in Groovy
The PR in question is here:
https://travis-ci.org/nextflow-io/nextflow/jobs/496572185
...
 Task :nextflow:processTestResources
> Task :nextflow:testClasses
> Task :nextflow:test
nextflow.executor.LsfExecutorTest > testHeaders FAILED
    org.spockframework.runtime.ConditionFailedWithExceptionError at LsfExecutorTest.groovy:71
        Caused by: groovy.lang.MissingPropertyException at LsfExecutorTest.groovy:71
nextflow.executor.LsfExecutorTest > testDiskResources FAILED
    groovy.lang.MissingPropertyException at LsfExecutorTest.groovy:218
nextflow.executor.LsfExecutorTest > testPerJobMemLimit FAILED
    org.spockframework.runtime.ConditionFailedWithExceptionError at LsfExecutorTest.groovy:254
        Caused by: groovy.lang.MissingPropertyException at LsfExecutorTest.groovy:254
nextflow.executor.LsfExecutorTest > testWorkDirWithBlanks FAILED
    org.spockframework.runtime.ConditionFailedWithExceptionError at LsfExecutorTest.groovy:313
        Caused by: groovy.lang.MissingPropertyException at LsfExecutorTest.groovy:313
1570 tests completed, 4 failed, 21 skipped

I know why the following needs to be changed:

https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/test/groovy/nextflow/executor/LsfExecutorTest.groovy#L71-L83

But I'm not sure how to easily check what the current code outputs without using println() statements....that feels a bit clunky

Laurence E. Bernstein
@lebernstein
Feb 22 23:56
using workflow.onComplete i would like to create a file that records the success/failure of the pipeline so that another process can wait for and read the file to know the outcome, however publishDir doesn't work inside the onComplete process, in fact I can not find any trace of a file created inside onComplete is there a way to do this?