Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
  • 13:44
    pditommaso commented #3367
  • 13:42
    marcodelapierre commented #3012
  • 13:35
    pditommaso milestoned #3012
  • 13:35
    pditommaso closed #3012
  • 13:35
    pditommaso commented #3012
  • 13:33
    marcodelapierre commented #3012
  • 13:30
    pditommaso milestoned #3472
  • 13:06
    phue commented #3367
  • 13:05
    phue commented #3367
  • 13:01
    marcodelapierre commented #3012
  • 13:01
    marcodelapierre opened #3478
  • 12:33
    l-modolo commented #3367
  • 11:59
    phue commented #3367
  • 11:56
    phue commented #3367
  • 11:55
    phue commented #3367
  • 11:52
    phue commented #3367
  • 11:43
    l-modolo commented #3367
  • 11:38
    phue commented #3367
  • 11:32
    phue commented #3367
  • 11:09
    pditommaso commented #3012
Phil Ewels

The gitter slack description points to it too:

Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html

You'll almost certainly get a better chance of picking up some help / responses there.. :+1:

Hey Guys,
Appreciate your help
I am facing an issue while using storeDir with s3 bucket.
We are using AWS EKS for the cluster, I use the storeDir directive together with an AWS s3 path to store the results.
If we switch to use publishDir instead of storeDir with the same bucket, I have no issues, which shows that it's not a local permissions issue. However, publishDir doesn't let us use the cached results.

I am getting the following error in my pipeline

Missing output file(s) `L1.Aligned.sortedByCoord.out.bam` expected by process `alignment`
  • Here is the process configuration

    params.results = “s3://k8s-temporal-efs/results/”
    process alignment {
    echo true
    tag { sample_name + “:” + lane_id }
    container ‘183449805178.dkr.ecr.us-east-1.amazonaws.com/nextflow-container:latest’
    pod = [nodeSelector: “${config.instanceType.alignment}, uuid=${params.uuid}“]
    cpus “${config.clustercores.alignment}” 
    // limit here to avoid overloading the S3 download pool
    maxForks 16
    maxRetries 3
    errorStrategy { sleep(Math.pow(2, task.attempt) * 15 as long); return ‘retry’ }
    storeDir { “${params.results}/$runid/$sample_name/align/$lane_id” }
    //publishDir { “${params.results}/$runid/$sample_name/align/$lane_id” }
    file reference
    set sample_name, sample_id, lane_id, file(read1), file(read2) from fastqs
    //val k from kickoff.first()
    val k from kickoff
    set sample_name, file (“${lane_id}.Aligned.sortedByCoord.out.bam”) into alignments
    set sample_name, file (“${lane_id}.Chimeric.out.sam”) into fusion_sams
    set sample_name, file (“${lane_id}.Chimeric.out.junction”) into fusion_jxns
    set sample_name, file (“${lane_id}.Log.final.out”) into star_logs
    set sample_name, sample_id, lane_id into meta
    template “alignment.sh”
  • Here is the error in .nextflow.log
    Process `alignment (A09-00159428-B-RNA-013:L1)` is unable to find [S3Path]: `/k8s-temporal-efs/results/210811_NB501495_0936_AH555LBGXK/A09-00159428-B-RNA-013/align/L1/L1.Aligned.sortedByCoord.out.bam` (pattern: `L1.Aligned.sortedByCoord.out.bam`)
Sam Birch
Is it possible to disable Singularity containers for one specific process if it has been enabled in nextflow.config? Most of our processes run inside Singularity containers using LSF, but I have one that needs to run locally using a conda env, and the conda setting seems to be ignored if Singularity is enabled.
1 reply
Does anyone know whether nextflow channels copy the data from one process to another or if the memory is shared somehow?
Hi all, a question about Nextflow on Kubernetes. The documentation says it requires a shared PVC for all pods, but it's unclear/conflicting on what that PVC is used for. Is it only for logs and such, and pods use transient storage for local processing, or is it also used for all local storage? I.e. if I have 3Tb worth of data, should I provision at least 3Tb of PVC (or look into other solutions entirely)?
Suchita Bhalerao
Unable to run rnaseq on AWS ..
Used this also but now it failed again ..
ERROR: Cannot download nextflow required file -- make sure you can connect to the internet
Suchita Bhalerao
WARN: Unable to start plugin 'nf-amazon' required by s3://xyz/runs/rnaseq/input/parameters.json............ Missing plugin 'nf-amazon' required to read file: s3://xyz/runs/rnaseq/input/parameters.json
Phil Ewels
Hi everyone! This Nextflow Gitter is now depreciated and has been replaced by the new Nextflow Slack. You can get an invite at https://www.nextflow.io/slack-invite.html
Austin Vecchio
Is there a way to fail a nextflow workflow run if it is not posting properly to a weblog server?
Princesca Dorsaint
hello I am running picards samtofastq to convert my bams to fastq file. I have saved the fastqs to a channel and would like to use them (R1 and R2) in the next process. Is there a way to use .fromFilePairs on a channel output?
Phil Ewels
@avecchio / @princescad:matrix.org / @PrincescaDorsaint - please ask your questions on the Nextflow Slack. This Gitter chat is no longer active. You can get an invite to the new Slack here: https://www.nextflow.io/slack-invite.html

params.out = "/Nextflow_test/results"

// publishDir "$params.out/$sample_id", mode: 'move'

set pair_id, file(reads) from trim_ch2
    file(STARgenome) from genomeIndex

    set pair_id, file("${pair_id}/*.out.bam") into bam_ch


    def output = "${pair_id}"

mkdir STAR_out

cd STAR_out

    STAR --genomeDir ${STARgenome} \
            --readFilesIn  ${reads} \
            --outFileNamePrefix ${pair_id} \
            --runThreadN ${task.cpus} \
            --twopassMode Basic \
            --outSAMunmapped Within \
            --chimSegmentMin 12 \
            --chimJunctionOverhangMin 12 \
            --alignSJDBoverhangMin 10 \
            --alignMatesGapMax 100000 \
            --chimSegmentReadGapMax 3 \
            --outFilterMismatchNmax 2 \
            --outSAMtype BAM SortedByCoordinate \
            --quantMode TranscriptomeSAM \
            --outBAMsortingThreadN 6 \
            --limitBAMsortRAM 80000000000
    mkdir ${output}
    mv *Aligned* ${output}
Caused by:
File /vf/users/Nextflow_test/work/8d/c23a34b1464c092dedf24d4b9b4d21/trim_Sample_test_R1.fastq/*.out.bam is out of the scope of process working dir: /vf/users/Nextflow_test/work/24/35c5bc4d2ac31651398a587f2b2cac. I am new to Nextflow, can someone help me understand the reason for this error.
Hi all, my nextflow script seemed to have ate my files.
I set a publish directory but I do not see anything in folder but the code was running without an error
does NF just ignore not being able to write to a directory
Bruce Moran
I'm trying to run on a cluster with Slurm, using salloc to allocate resources, and then srun NF in backgound mode srun nextflow <pipeline> -bg > log.txt. This doesn't work, no job is submitted by NF. Can someone tell me if they've done this, how?
1 reply
Weinong Han
Hello, I am new to nf-core/pipeline and failed to pull singularity image

WARN: Singularity cache directory has not been defined -- Remote image will be shan/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a d

Caused by:
Failed to pull singularity image
command: singularity pull --name depot.galaxyproject.org-singularity-sdrf-pip0.img.pulling.1657154892314 https://depot.galaxyproject.org/singularity/sdrf-pip0 > /dev/null
status : 255
ERROR: pull is only supported for shub URIs

Nurlan Kerimov

Hello. Help needed! probably some of you already had it but I could not find it in history:
when I get a single file from S3 using fromPath it works. However when I use glob pattern it throws

Cannot find nmr_data in: s3://my_bucket/fi*.json


Channel.fromPath(params.nmr_data, type: "any")
    .ifEmpty { error "Cannot find nmr_data in: ${params.nmr_data}"}
    .view( {it.text} ) //{file(it).text }
    .set { nmr_data }

nextflow run main.nf --nmr_data "s3://my_bucket/file.json" // THIS WORKS
nextflow run main.nf --nmr_data "s3://my_bucket/fi*.json" // THIS DOES NOT WORK
Daniel Malzl


I am currently trying to port one of our pipelines to DSL2 but fail to do so due to an issue with the output routing which is beyond my comprehension.
In particular I have some fastq files which I'd like to trim with trim_galore for which reason I coded the following module:

process TRIM_GALORE {

    tag "$meta.id"

    tuple val(meta), path(reads)

    tuple val(meta), path("*_trimmed_val_*.fq.gz"), emit: reads
    path "*fastqc.{zip,html}",                      emit: fastqc
    path "*trimming_report.txt",                    emit: reports

    read1       = reads[0]
    read2       = reads[1]
    lastPath    = read1.lastIndexOf(File.separator)
    read1Base   = read1.substring(lastPath+1)
    lastPath    = read2.lastIndexOf(File.separator)
    read2Base   = read2.substring(lastPath+1)

    trim_galore --paired \
                --quality 20 \
                --fastqc \
                --illumina \
                --gzip \
                --output_dir !{meta.id} \
                --basename !{meta.id}_trimmed \
                --cores !{task.cpus} \
                !{read1} \

    mv !{read1Base}_trimming_report.txt !{meta.id}_trimmed_val_1.fq.gz_trimming_report.txt
    sed -i 's/Command line parameters:.*\$/Command line parameters: !{meta.id}_trimmed_val_1/g' !{meta.id}_trimmed_val_1.fq.gz_trimming_report.txt
    mv !{read2Base}_trimming_report.txt !{meta.id}_trimmed_val_2.fq.gz_trimming_report.txt
    sed -i 's/Command line parameters:.*\$/Command line parameters: ${meta.id}_trimmed_val_2/g' !{meta.id}_trimmed_val_2.fq.gz_trimming_report.txt

The corresponding workflow composition looks like something like this:

    // read QC
    TRIM_GALORE ( ch_cat_fastq )

    // splitting fastqs for HICUP parallelization
    SPLIT_FASTQ ( TRIM_GALORE.out.reads )

However, regardless of what I do I keep getting a No such variable: Exception evaluating property 'out' for nextflow.script.ChannelOut error. If I remove the out. I just get No such variable: reads

I couldn't find anything regarding this on the internet. Does anyone have a clue?

Daniel Malzl
Nextflow version is 21.10.0
Christian Tischer
Hi all, I am using nextflow to launch processes that rely on Java.
Processing them in parallel gives me Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
When executing them sequentially (executor.queueSize = 1) I don't get this error. Does anyone know how to fix this?
Karin Lagesen
hi all
I am wondering how I can specify options to a process outside of the process itself, that is, in a config file
specifically, I want to put the conda directive into a config file
one per process
note, I am still on dsl1

I am trying to run TransPi with the '--all' option, but the process halted abruptly stating the following error:

DEBUG nextflow.Session - Session aborted -- Cause: Operator into has been deprecated -- it's not available in DSL2 syntax
Jul-22 14:24:15.801 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.DeprecationException: Operator into has been deprecated -- it's not available in DSL2 syntax
I assume this is an issue with the new nextflow update. Is there a way to fix this?

Hey all, you'll probably get better support on Slack than here:
Hi, is it possible to debug the nextflow sscripts with a debugger?
Hi there! How many possible combinations can be obtained for the "Run Name" between the two word sets? And how big is each set? Thanks!
Hussein Elgridly
Hi folks - quick Q. If I have a file in a versioned S3 bucket, can I refer to a non-latest version of it in Nextflow? e.g. s3://bucket/my.file?version=12345
@BonaBeavis @santiagorevale @helgridly You may want to try the Slack community, this gitter really isn't active any more.
"I'll help 10individuals how to earn $30,000 in 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message on Telegram by asking me (HOW) for more details on how to get started
Stijn van Dongen
It's been a while since I visited ... hello dear NF hackers! I'm puzzling over which Java is best to install on macos/M1 architecture (with the purpose of running NF of course). I see Oracle, there's brew cask, or even conda ... any recommendations/considerations?
Stijn van Dongen
I went with oracle aarch64 java, then hit an issue described in #2856, which was solved by the last comment's suggestion export JAVA_HOME=$(/usr/libexec/java_home -v 18)
Is this still a good place to hang around? Or is there a better place?
Moritz E. Beber
Stijn van Dongen
:+1: :+1: :+1:
(I now see a few lines above Paolo posted the same :facepalm:
Win up to $1000 in crypto trading when you invest with just the minimum of $100.
Signup and start investing your crypto with.
i am having trouble understanding REVISION ID in the nextflow execution log (.nextflow/history). the documentation says it should be a git branch/tag/commit SHA, but i see only a long string that does not appear anywhere in my git log. is the documentation correct?
Anand Mayakonda
Hello all. How do I sort the input files by reverse order i.e reverse-lexicographic-order ?
I have this command which lists all vcf files in lexicographic order SNV_vcf_files = file("03_snvs/*.hard-filtered.vcf.gz", sort: true) I want to sort them in reverse order.
Anand Mayakonda
I just added reverse tag. SNV_vcf_files = file("03_snvs/*.hard-filtered.vcf.gz", sort: true).reverse() Do let me know if there is an elegant way of doing this.