Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:24
    pditommaso commented #1952
  • 10:21

    pditommaso on nf-sqldb

    Fix minor typo [ci skip] Fix output unstage with glob pa… Refactor AWS batch job def crea… and 55 more (compare)

  • 01:15
    fujimuramasa edited #2248
  • Jul 29 23:11
    benjschiller commented #1952
  • Jul 29 17:34

    pditommaso on master

    Restore debug line for cache he… (compare)

  • Jul 29 15:32
    yangxiao1220 commented #2249
  • Jul 29 15:21
    yangxiao1220 opened #2249
  • Jul 29 14:30

    pditommaso on master

    Fix Prevent script var names co… (compare)

  • Jul 29 14:09
    abhi18av labeled #2246
  • Jul 29 14:09
    abhi18av unlabeled #2246
  • Jul 29 14:06
    Lehmann-Fabian review_requested #2182
  • Jul 29 13:49
    abhi18av labeled #2246
  • Jul 29 13:49
    pditommaso commented #2246
  • Jul 29 13:49
    Midnighter commented #2246
  • Jul 29 13:27
    pditommaso commented #2246
  • Jul 29 12:11

    pditommaso on master

    Improve error reporting [ci fas… (compare)

  • Jul 29 11:21
    Midnighter commented #2246
  • Jul 29 10:29
    abhi18av labeled #1670
  • Jul 29 10:26
    wbazant commented #1670
  • Jul 29 08:59
    abhi18av labeled #2246
Sam Choi
@choishingwan_gitlab
this is strange as 1. I didn't actually return the result from the second map (I return the original value a) 2. I did not assign the map to anything
so it is unclear to me why the intermediate of the second map is being appended to the first map
does that have something to do with variable initiation?
namiller2015
@namiller2015

Hi All,
I'm a bit confused on Nextflow's interaction with AWS credentials.
I'm getting the following error when a docker image does NOT exist locally on my EC2 already

Unable to find image 'AWS S3 URI link to my ECR docker repo' locally
docker: Error response from daemon: Get 'AWS S3 URI link' no basic auth credentials.

If I pull the docker image before starting nextflow I don't get this error. However, I know nextflow is able to interact correctly with the AWS API because nextflow is reading and writing to S3 buckets with no issue.

I don't quite understand why this is an issue with docker images in my ECR. I will say I did not set up the infrastructure for the AWS IAM roles. But I'm not assuming a new role when pulling the docker image outside of nextflow and that doesn't throw an error.

This isn't a big deal when running on a single EC2 instance. I can pull all the docker images I need before starting nextflow. But I am going to run in AWS Batch soon and I'm worried nextflow won't be able to pull docker images for each compute node.

Thanks!

daudn
@daudn

Hello Nextflow - quick question.

I am using Nextflow to run a hybrid workflow (Local + GCP Cloud)

Other than running the workflow, I also need to backup the initial raw data.

I have tried doing this with Nextflow, although nextflow doesn't let me output files which are the same as the input files (aka if they have not been processed/altered, nextflow doesn't accept it)

Is there any clean way of doing this?

1 reply
Ideally when the workflow completes, I would like Nextflow to upload the initial raw data to a bucket of my choosing.
Joan Gaztelu
@joanGaztelu_gitlab
Hello I am building a pipeline and I have some doubts about file implementations. In a specific process I execute a software command and in some paramaters, I would like to introduce a file as a parameter (this way, the software reads a file and not a string typed in the command line). The things is, I want the current process to receive the output from previous steps and also use this external file as a parameter. How do I introduce it in the pipeline? Thanks
Laurence E. Bernstein
@lebernstein

@joanGaztelu_gitlab
If I am understanding your question you want to do something like:

externalFile = file(params.externalFile)

processX {
}

processY {
   processX.out,
  externalFile
}

This is DSL2 syntax

Steven P. Vensko II
@spvensko_gitlab

Is storeDir incompatible with the google-lifesciences executor?

This block:

process trim_galore_exec {

  label 'trim_galore_container'
  label 'trim_galore'
  publishDir "${proj_dir}/${uniq_dir.getText()}"
  storeDir "${shared_dir}/${uniq_dir.getText()}"
  tag "${dataset}/${pat_name}/${prefix}"

results in a NOTE: Cannot access folder: '/path/to/my/storeDir/output' error

but if I modify it with:

process trim_galore_exec {

  label 'trim_galore_container'
  label 'trim_galore'
  publishDir "${shared_dir}/${uniq_dir.getText()}"
//  storeDir "${shared_dir}/${uniq_dir.getText()}"
  tag "${dataset}/${pat_name}/${prefix}"

Then the publishDir directive publishes the output files produced in GCP in the expected "shared" directory. This seems to indicate storeDir is incompatible with google-lifesciences, but I don't see anything suggesting this in the documentation.

1 reply
Rory Kirchner
@roryk
Hey all, I have spent hours trying to figure this out by myself. I have a channel that's emitting tuples of [samplename, path] and want to call a python program with flags like --samples comma-separated-samplenames --paths comma-separated-paths-to-those-files. I can't figure out how to have this happen using the DSL 2 syntax. If someone could point me to an example or some relevant documentation that would be amazing. Thanks for all your work.
6 replies
Rory Kirchner
@roryk
Ok I think I can get there if I stop emitting a tuple and just emit to two different channels.
3 replies
Aaron Statham
@astatham
Hi - does anyone have any tips when publishing large output files generated on AWS Batch to S3? When publishing a 1.2TB BAM, nextflow outputs Waiting files transfer to complete (1 files) for over a day then crashes with a Java heap error (even when I increase the heap size to 150GB). I just filed this bug report nextflow-io/nextflow#2230 but maybe someone knows some parameters I can test tweaking in the meantime?
2 replies
🇮🇳Ajay Mittal🇮🇳
@ajaymittal81_twitter
Hi, I am trying to call another bash script (workflow) from the another script (workflow). I tried afterScript, script directive but getting error every time. Is it possible in Nextflow to chain the workflows using aftrScript/script directive? If it is, could you pls help me the code snippet.
1 reply
Dagney Taggart
@dagneytaggartsv_twitter

All, I have a simple question: how do I establish dependencies between workflows? In the following snippet "data.value" is always true regardless of w1 succeeding or failing.

```workflow w1 {
p1 | (p2 & p3)
emit:
true
}

workflow w2 {
take:
data
main:
println("Did w1 succeed?:" + data.value);
if (data.value) {
p4 | p5
}
}

workflow {
w1 | w2
}```

4 replies
Adam Talbot
@adamrtalbot

I see there are parameters for accessing a container registry in the Azure documentation (21.06.0), but when I add them to my config it raises the error One or more container images specified are invalid. The relevant part of the config is (fake details):

plugins {
    id 'nf-azure@0.10.0'
}

azure {
    batch {
        registry {
            server = "example.azurecr.io"
            userName = "example_user"
            password = "password"
        }
    }
}

I've also tried under a profile (azure.batch.registry) but it doesn't seem to change anything. When I investigate the tasks, there is no Registry Server or Registry Username configured per task. Can anyone see where I'm going wrong?

KyleStiers
@KyleStiers

Is it possible to use .unique() channel operator on a tuple? I have an input channel that accepts a tuple like:

[id, project_name1]
[id, project_name1]
[id, project_name2]
[id, project_name2]

I'd like to accept this channel and keep only the unique tuples, i.e.

[id, project_name1]
[id, project_name2]

I'm trying to do this with a simple chaining of .collect().unique() but can't figure out if a tuple is possible. When I flatten it instead of collect, unique works, but then I lose the structure of the tuple and groupTuple() only sort of works to put it back together. It would be wonderful if there was a way to make .unique() work on tuples.

Matthew Nguyen
@matnguyen
What's the best way for me to import both the Nextflow processes as well as the Python scripts from one Nextflow workflow to another?
I've looked into Git submodules, but I couldn't import the Python scripts from the submodule into my main Python scripts in bin
I've also looked into packaging the original Nextflow workflow as a docker container, but if I change the namespace so I don't import the previous Python scripts as bin.x, then Nextflow can't call these scripts since it only adds bin to the path
mmatthews06
@mmatthews06
Question: Is there a way to exit a Nextflow pipeline with an error, and actually inform NF-Tower, or send a weblog message? I've tried exit 1, error(message) and throwing an Exception, and none of them have worked well.
2 replies
Brandon Cazander
@brandoncazander

We sometimes (spuriously) have a large number of nextflow processes on AWS Batch that will fail to start with DockerTimeoutError: Could not transition to created; timed out after waiting 4m0s. Sometimes retrying the process works, but usually it will run out of retries (maxRetries: 2) and the whole job will fail.

Has anyone ever run into this and fixed it?

7 replies
LehmannN
@LehmannN

Question: How can I run multiple nextflow pipelines in parallel without getting the error "Unable to acquire lock on session with ID xxx" ? I would like to run multiple times the same pipeline simultaneously on a cluster (with different parameters) but I get stopped with this error and couldn't find how to get through this. Thanks for the help !

Ps: I observed that the name of the workflow is different each time, but not the ID. Example:
Launching main.nf [naughty_linnaeus] - revision: 8a3938ec44
Launching main.nf [elated_solvay] - revision: 8a3938ec44

LehmannN
@LehmannN

Different question than the one I posted earlier today, but same goal: still to run multiple pipelines simultaneously. Now I've tried to run it using the HTcondor executor. For each run, I have different arguments, and all the arguments are in one file. Example:

> cat arguments.tsv
param1 param2 param3
param1 param2 param3

So each line corresponds to a single run.
I tried a solution with xargs to run it, such as xargs -L1 -a params_full.tsv ./_run_nextflow.sh, which works in a sense, but each pipeline runs one after the other, not simultaneously.

My script inside _run_nextflow.sh is :

nextflow run main.nf -profile condor \
        --workflow reference \
        --gtf $1 \
        --feature $2 \
        --outDir $3

with $1 = param1, $2 = param2, etc.
Any idea on that ?

Chenhao Li
@lch14forever
Hi, is that possible to let workflow.onComplete call system command? I am wondering if I can send a post request using cURL on completion/error.
2 replies
philDTU
@philDTU
Is it possible to use a singularity container only in a one specific process out of many?
4 replies
namiller2015
@namiller2015

Hi All,
I'm getting an error running nextflow on an AWS EC2 instance when the docker image is not already downloaded to the local EC2.

no basic auth credentials.

I know nextflow can access my AWS environment because it can read and write to a S3 bucket. Any ideas on why nextflow can't pull a docker image from my AWS ECR?
Thanks!

Enorya
@Enorya

Hi! The use of Nextflow is new for me and I'm trying to concatenate multiple fastq files into one big fastq file. I tried to use the same technique as in the example https://github.com/nextflow-io/patterns/blob/master/docs/collect-into-file.adoc like this:

Channel.fromPath('coral-sequences/*.fastq').set { coralReads_ch }

process mergeReadFiles {
  publishDir './Results', mode: 'copy', overwrite: false
  memory '1 GB'
  time '1h'
  cpus 1
  clusterOptions '-A llp_lbeg'

  input:
  file fastq from coralReads_ch

  output:
  file 'all_coral_reads.fastq' into allReads_ch

  script:
  """
  < $fastq cat > all_coral_reads.fastq
  """
}

But I don't end up with what I want (meaning the big fastq file in the publishDir) and I have the following error:

Jul-27 10:46:55.575 [Actor Thread 1] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /local_scratch/074bc15d79b77f873b4a4ae7e4d59232.collect-file
Jul-27 10:46:55.738 [Actor Thread 1] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /local_scratch/nxf-7697929815299128940
Jul-27 10:46:57.416 [Actor Thread 44] ERROR nextflow.extension.DataflowHelper - @unknown
java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at org.codehaus.groovy.runtime.IOGroovyMethods.getText(IOGroovyMethods.java:862)
    at org.codehaus.groovy.runtime.NioGroovyMethods.getText(NioGroovyMethods.java:423)
    at sun.reflect.GeneratedMethodAccessor668.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
    at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1859)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3797)
    at groovy.lang.DelegatingMetaClass.getProperty(DelegatingMetaClass.java:130)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:195)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassGetPropertySite.getProperty(PojoMetaClassGetPropertySite.java:36)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
    at _nf_script_470ae485$_run_closure3.doCall(_nf_script_470ae485:26)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)

Can you help me?

Paolo Di Tommaso
@pditommaso
I don't think the error is related to the above snippet or at least is not complete
Moritz E. Beber
@Midnighter

Can anyone explain to me why the following doesn't work (Error: No such variable: meta):

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process FOO {
  input:
  val(meta)

  output:
  tuple val(meta), path(meta.id), emit: reads

  script:
  """
  mkdir "${meta.id}"
  """
}

workflow {
    FOO(Channel.of([id: 'sequence1']))
}

but this does, i.e., a meaningless change in how the path is specified:

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process FOO {
  input:
  val(meta)

  output:
  tuple val(meta), path("${meta.id}"), emit: reads

  script:
  """
  mkdir "${meta.id}"
  """
}

workflow {
    FOO(Channel.of([id: 'sequence1']))
}
Appreciate any insights, thanks :slight_smile:
Moritz E. Beber
@Midnighter
Another one that I couldn't find documentation on anywhere: It seems using meta as a channel name is not allowed to be passed to a process.
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process FOO {
  input:
  val(meta)

  """
  touch "${meta.id}"
  """
}


workflow {
  meta = Channel.of([id: 'sequence1'])
  FOO(meta)
}
Cannot cast object 'DataflowBroadcast around DataflowStream[?]' with class 'groovyx.gpars.dataflow.DataflowBroadcast' to class 'nextflow.script.ScriptMeta'

Or alternatively:

meta = Channel.of([id: 'sequence1'])

workflow {
  FOO(meta)
}

leads to

Error executing process > 'FOO'

Caused by:
  No such property: id for class: nextflow.script.ScriptMeta

Source block:
  """
    touch "${meta.id}"
    """
KyleStiers
@KyleStiers

Not a terribly important question but I'm curious. Is it possible to configure nextflows interactive reporting (the stdout stuff). For instance:

[4a/fd89aa] process > process 1 (2)  [  0%] 0 of 5
[34/517886] process >  process 2 [100%] 1 of 1 ✔
[-        ] process > process 3  -
[-        ] process >  process 4 -

So that the [xxx%] N of M part is aligned to the right side to permit more space to view the tags you may assign for each process? Would be kind of neat to be able to play with this

Enorya
@Enorya
Hi @pditommaso, yes sorry the error wasn't complete (It's really ling and I thought everything was copied)
java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.read1(BufferedReader.java:210)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    at java.io.Reader.read(Reader.java:140)
    at org.codehaus.groovy.runtime.IOGroovyMethods.getText(IOGroovyMethods.java:862)
    at org.codehaus.groovy.runtime.NioGroovyMethods.getText(NioGroovyMethods.java:423)
    at sun.reflect.GeneratedMethodAccessor668.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
    at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1859)
    at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3797)
    at groovy.lang.DelegatingMetaClass.getProperty(DelegatingMetaClass.java:130)
    at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:195)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassGetPropertySite.getProperty(PojoMetaClassGetPropertySite.java:36)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
    at _nf_script_470ae485$_run_closure3.doCall(_nf_script_470ae485:26)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at nextflow.extension.DataflowExtensions$_println_closure45.doCall(DataflowExtensions.groovy:1535)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at groovy.lang.Closure$call$9.call(Unknown Source)
    at nextflow.extension.DataflowHelper$_subscribeImpl_closure2.doCall(DataflowHelper.groovy:231)
    at sun.reflect.GeneratedMethodAccessor686.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at groovy.lang.Closure$call$9.call(Unknown Source)
    at nextflow.extension.DataflowHelper$_subscribeImpl_closure2.doCall(DataflowHelper.groovy:231)
    at sun.reflect.GeneratedMethodAccessor686.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:104)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:326)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at groovy.lang.Closure.call(Closure.java:411)
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:108)
    at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
    at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
    at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
    at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
    at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
This is the complete error I get
Darek K.
@darked89
Hi. I am translating a Python script based pipeline to the Nextflow DSL 2. My idea us to pinch out the parameters, like input files, output dir etc. and have it in YAML/TOML. But maybe the better way would be to switch to the Nextflow config format and test it with Python pipeline first. Hence my questions: 1) does it make sense? 2) is there some Python module to handle Nextflow config format? Thanks a lot for your help
Sosi
@Chisosi_gitlab
Hi, can anyone tell me any way I can check what is happening with my run? nextflow log shows that DURATION is empty (i.e. it didnt start), and when I check the log it seems it is hung somewhere with no error or warning. I've used nohup nextflow run ...
it has been over 30 mins since i asked the run to start
any idea of how to even debug this?
I can see that the work/ folder was created, but it is empty
Sosi
@Chisosi_gitlab
when I relaunch with nohup nextflow run -resume ... it tells me it is already being run
1 reply
Yanick Hagmeijer
@HagemeijerRug

I have implemented a feedback look in DSL2 using code from this github issue

as long as I can somehow hint nextflow that the feedback channel ends nextflow runs correctly:

feedback_subset = feedback_channel.take( 1 )
DNA_like_bams_to_calibrate = DNA_like_bam.mix(feedback_subset)

However, I don't want to hardcode the 1. I want it to be based on the .count() value from a different channel that is not involved in the loop. But whenever it anything like that that I get:
Missing process or function with name 'take'

can someone explain how I can get the count/size of a channel that has already finished to be used as an argument for such a function?

BarryDigby
@BarryDigby

Struggling with filter() here if anyone has some insights:

#!/usr/bin/env nextflow 

process make_bt{

    output:
    file("chrI.*") into ch_bowtie

    script:
    """
    touch chrI.fa
    touch chrI.{1..5}.bt2
    """
}

process make_fasta{

    output:
    file("chrI.fa") into ch_fasta

    script:
    """
    touch chrI.fa
    """
}

ch_avoid_collisions = ch_bowtie.filter{ it.endsWith("/bt2/") } 
ch_avoid_collisions = ch_bowtie.filter{ it.toString().endsWith("/bt2/") }
ch_avoid_collisions = ch_bowtie.filter{ it =~ /bt2/ }

process find_circ{

   echo true 

   input:
   file(fasta) from ch_fasta
   file(bowtie_index) from ch_avoid_collisions.collect()

   output:
   stdout to out

   script:
   """
   ls -la *
   """
}

I've included failed attempts in ch_avoid_collision to help you see what I'm getting at. Any help is greatly appreciated! :)

1 reply
Robin F Chan
@robinfchan

Re-working some stuff for DSL2 and a bit a lost why I cannot get Nextflow to spin-up multiple processes for each input file. When I execute the workflows it starts up fine but is only starting a process for the first file:

(base) [ec2-user@ip-10-58-22-151 fs1]$ nextflow /mnt/efs/fs1/code/nanosuite/main.nf --reads "/mnt/efs/fs1/data/nanopore/toy/FAQ*.fastq.gz" --mmi "/mnt/efs/fs1/code/nanosuite/assets/LIB1.mmi" --outdir
"/mnt/efs/fs1/nextflow/" -profile "docker"
N E X T F L O W  ~  version 21.04.3
Launching `/mnt/efs/fs1/code/nanosuite/main.nf` [marvelous_mahavira] - revision: 5fc96f1734
Run Name          : marvelous_mahavira
Reads             : /mnt/efs/fs1/data/nanopore/toy/FAQ*.fastq.gz
Minimizer Index   : /mnt/efs/fs1/code/nanosuite/assets/LIB1.mmi
Max Resources     : 128.GB memory, 16 cpus, 240.h time per job
Container         : docker - [:]
Output Dir        : /mnt/efs/fs1/nextflow/
Launch Dir        : /mnt/efs/fs1
Working Dir       : /mnt/efs/fs1/work
Script Dir        : /mnt/efs/fs1/code/nanosuite
User              : ec2-user
Config Profile    : docker
----------------------------------------------------
executor >  local (1)
[c0/03beee] process > NANOSUITE_ALIGN_POOLED_SCREEN:ALIGN_POOLED_SCREEN:MINIMAP2_ALIGN (FAQ17884_pass_76111b32_1) [  0%] 0 of 1
[-        ] process > NANOSUITE_ALIGN_POOLED_SCREEN:ALIGN_POOLED_SCREEN:SAMTOOLS_SORT

Here are my channels, am I doing something wrong?

workflow ALIGN_POOLED_SCREEN {

    ch_reads = Channel.fromPath( params.reads ).map{ file -> tuple(file.simpleName, file) }
    ch_mmi = Channel.fromPath( params.mmi )

    MINIMAP2_ALIGN( ch_reads, ch_mmi )

    ch_samtools_sort_input = MINIMAP2_ALIGN.out.bam.map{ file -> tuple(file.simpleName, file) }
    SAMTOOLS_SORT( ch_samtools_sort_input )
}

and the input files in question:

(base) [ec2-user@ip-10-58-22-151 fs1]$ ls -l /mnt/efs/fs1/data/nanopore/toy/
total 24296
-rwxrwx--- 1 ec2-user ec2-user 12439037 Jul 29 15:53 FAQ17884_pass_76111b32_0.fastq.gz
-rwxrwx--- 1 ec2-user ec2-user 12439037 Jul 29 15:53 FAQ17884_pass_76111b32_1.fastq.gz
Robin F Chan
@robinfchan
UUghhhhhh nevermind. I forgot to declare ".collect()" after the ch_mmi since it is a re-used file 😔
Matthew Nguyen
@matnguyen

How do I input a path to a directory for a nextflow process? I get this error:

Caused by:
  Not a valid path value: assets/models/

where in my process I have

input: 
    path trained_models_dir
    path fitted_scalers_dir

and my config file has:

params.trained_models_dir = "assets/models"
params.fitted_scalers_dir = "assets/scalers"
2 replies
Alex Mestiashvili
@mestia
Is it possible to change the queueSize or any other pipeline parameters while it is running?
Nine-s
@Nine-s

Hello! I am trying to split FASTQ files obtained from the fromSRA method. Does anyone know how to do that?

read_pairs_ch = Channel.fromSRA( "SRR3287149" , apiKey:'mykey').splitFastq(by:50, pe:true, decompress:true, file:true).view()

I get

Cannot find splittable file (elem=-2)


Cannot find splittable file (elem=-1)
1 reply
Dagney Taggart
@dagneytaggartsv_twitter
Can I pass environment variables to nextflow via command line like this: nextflow run workflow.nf --env.PYTHONPATH=somepath?
1 reply