Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 13:44
    pditommaso commented #3367
  • 13:42
    marcodelapierre commented #3012
  • 13:35
    pditommaso milestoned #3012
  • 13:35
    pditommaso closed #3012
  • 13:35
    pditommaso commented #3012
  • 13:33
    marcodelapierre commented #3012
  • 13:30
    pditommaso milestoned #3472
  • 13:06
    phue commented #3367
  • 13:05
    phue commented #3367
  • 13:01
    marcodelapierre commented #3012
  • 13:01
    marcodelapierre opened #3478
  • 12:33
    l-modolo commented #3367
  • 11:59
    phue commented #3367
  • 11:56
    phue commented #3367
  • 11:55
    phue commented #3367
  • 11:52
    phue commented #3367
  • 11:43
    l-modolo commented #3367
  • 11:38
    phue commented #3367
  • 11:32
    phue commented #3367
  • 11:09
    pditommaso commented #3012
Phil Ewels
@ewels

The gitter slack description points to it too:

Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html

You'll almost certainly get a better chance of picking up some help / responses there.. :+1:
SHERIF
@syousry6

Hey Guys,
Appreciate your help
I am facing an issue while using storeDir with s3 bucket.
We are using AWS EKS for the cluster, I use the storeDir directive together with an AWS s3 path to store the results.
If we switch to use publishDir instead of storeDir with the same bucket, I have no issues, which shows that it's not a local permissions issue. However, publishDir doesn't let us use the cached results.

I am getting the following error in my pipeline

Missing output file(s) `L1.Aligned.sortedByCoord.out.bam` expected by process `alignment`
  • Here is the process configuration

    params.results = “s3://k8s-temporal-efs/results/”
    process alignment {
    echo true
    tag { sample_name + “:” + lane_id }
    container ‘183449805178.dkr.ecr.us-east-1.amazonaws.com/nextflow-container:latest’
    
    pod = [nodeSelector: “${config.instanceType.alignment}, uuid=${params.uuid}“]
    cpus “${config.clustercores.alignment}” 
    memory${config.clustermem.alignment}” 
    
    // limit here to avoid overloading the S3 download pool
    maxForks 16
    maxRetries 3
    errorStrategy { sleep(Math.pow(2, task.attempt) * 15 as long); return ‘retry’ }
    
    storeDir { “${params.results}/$runid/$sample_name/align/$lane_id” }
    //publishDir { “${params.results}/$runid/$sample_name/align/$lane_id” }
    
    input:
    file reference
    set sample_name, sample_id, lane_id, file(read1), file(read2) from fastqs
    //val k from kickoff.first()
    val k from kickoff
    
    output:
    set sample_name, file (“${lane_id}.Aligned.sortedByCoord.out.bam”) into alignments
    set sample_name, file (“${lane_id}.Chimeric.out.sam”) into fusion_sams
    set sample_name, file (“${lane_id}.Chimeric.out.junction”) into fusion_jxns
    
    set sample_name, file (“${lane_id}.Log.final.out”) into star_logs
    
    set sample_name, sample_id, lane_id into meta
    
    script:
    template “alignment.sh”
    }
  • Here is the error in .nextflow.log
    Process `alignment (A09-00159428-B-RNA-013:L1)` is unable to find [S3Path]: `/k8s-temporal-efs/results/210811_NB501495_0936_AH555LBGXK/A09-00159428-B-RNA-013/align/L1/L1.Aligned.sortedByCoord.out.bam` (pattern: `L1.Aligned.sortedByCoord.out.bam`)
Sam Birch
@hotplot
Is it possible to disable Singularity containers for one specific process if it has been enabled in nextflow.config? Most of our processes run inside Singularity containers using LSF, but I have one that needs to run locally using a conda env, and the conda setting seems to be ignored if Singularity is enabled.
1 reply
RonTuretzkyMigal
@RonTuretzkyMigal
Does anyone know whether nextflow channels copy the data from one process to another or if the memory is shared somehow?
Aleksandr
@aleksandrpetukhov
Hi all, a question about Nextflow on Kubernetes. The documentation says it requires a shared PVC for all pods, but it's unclear/conflicting on what that PVC is used for. Is it only for logs and such, and pods use transient storage for local processing, or is it also used for all local storage? I.e. if I have 3Tb worth of data, should I provision at least 3Tb of PVC (or look into other solutions entirely)?
Suchita Bhalerao
@suchita-bhalerao_psl
Unable to run rnaseq on AWS ..
NXF_PLUGINS_DEFAULT=nf-amazon@1.8.0
Used this also but now it failed again ..
ERROR: Cannot download nextflow required file -- make sure you can connect to the internet
Suchita Bhalerao
@suchita-bhalerao_psl
WARN: Unable to start plugin 'nf-amazon' required by s3://xyz/runs/rnaseq/input/parameters.json............ Missing plugin 'nf-amazon' required to read file: s3://xyz/runs/rnaseq/input/parameters.json
Phil Ewels
@ewels
Hi everyone! This Nextflow Gitter is now depreciated and has been replaced by the new Nextflow Slack. You can get an invite at https://www.nextflow.io/slack-invite.html
Austin Vecchio
@avecchio
Is there a way to fail a nextflow workflow run if it is not posting properly to a weblog server?
Princesca Dorsaint
@princescad:matrix.org
[m]
hello I am running picards samtofastq to convert my bams to fastq file. I have saved the fastqs to a channel and would like to use them (R1 and R2) in the next process. Is there a way to use .fromFilePairs on a channel output?
Phil Ewels
@ewels
@avecchio / @princescad:matrix.org / @PrincescaDorsaint - please ask your questions on the Nextflow Slack. This Gitter chat is no longer active. You can get an invite to the new Slack here: https://www.nextflow.io/slack-invite.html
vinegang
@vinegang

params.out = "/Nextflow_test/results"

// publishDir "$params.out/$sample_id", mode: 'move'

    input:
set pair_id, file(reads) from trim_ch2
    file(STARgenome) from genomeIndex

    output:
    set pair_id, file("${pair_id}/*.out.bam") into bam_ch

    script:

    def output = "${pair_id}"
    """

mkdir STAR_out

cd STAR_out

    STAR --genomeDir ${STARgenome} \
            --readFilesIn  ${reads} \
            --outFileNamePrefix ${pair_id} \
            --runThreadN ${task.cpus} \
            --twopassMode Basic \
            --outSAMunmapped Within \
            --chimSegmentMin 12 \
            --chimJunctionOverhangMin 12 \
            --alignSJDBoverhangMin 10 \
            --alignMatesGapMax 100000 \
            --chimSegmentReadGapMax 3 \
            --outFilterMismatchNmax 2 \
            --outSAMtype BAM SortedByCoordinate \
            --quantMode TranscriptomeSAM \
            --outBAMsortingThreadN 6 \
            --limitBAMsortRAM 80000000000
    mkdir ${output}
    mv *Aligned* ${output}
Caused by:
File /vf/users/Nextflow_test/work/8d/c23a34b1464c092dedf24d4b9b4d21/trim_Sample_test_R1.fastq/*.out.bam is out of the scope of process working dir: /vf/users/Nextflow_test/work/24/35c5bc4d2ac31651398a587f2b2cac. I am new to Nextflow, can someone help me understand the reason for this error.
mjr1ch
@mjr1ch
Hi all, my nextflow script seemed to have ate my files.
I set a publish directory but I do not see anything in folder but the code was running without an error
does NF just ignore not being able to write to a directory
Bruce Moran
@brucemoran
I'm trying to run on a cluster with Slurm, using salloc to allocate resources, and then srun NF in backgound mode srun nextflow <pipeline> -bg > log.txt. This doesn't work, no job is submitted by NF. Can someone tell me if they've done this, how?
1 reply
Weinong Han
@WNHAN
Hello, I am new to nf-core/pipeline and failed to pull singularity image

WARN: Singularity cache directory has not been defined -- Remote image will be shan/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a d
Error executing process > 'NFCORE_QUANTMS:QUANTMS:CREATE_INPUT_CHANNEL:SDRFPARSI

Caused by:
Failed to pull singularity image
command: singularity pull --name depot.galaxyproject.org-singularity-sdrf-pip0.img.pulling.1657154892314 https://depot.galaxyproject.org/singularity/sdrf-pip0 > /dev/null
status : 255
message:
ERROR: pull is only supported for shub URIs

Nurlan Kerimov
@kerimoff

Hello. Help needed! probably some of you already had it but I could not find it in history:
when I get a single file from S3 using fromPath it works. However when I use glob pattern it throws

UnsupportedOperationException
Cannot find nmr_data in: s3://my_bucket/fi*.json

Examples:

Channel.fromPath(params.nmr_data, type: "any")
    .ifEmpty { error "Cannot find nmr_data in: ${params.nmr_data}"}
    .view( {it.text} ) //{file(it).text }
    .set { nmr_data }

nextflow run main.nf --nmr_data "s3://my_bucket/file.json" // THIS WORKS
nextflow run main.nf --nmr_data "s3://my_bucket/fi*.json" // THIS DOES NOT WORK
Daniel Malzl
@dmalzl

Hi,

I am currently trying to port one of our pipelines to DSL2 but fail to do so due to an issue with the output routing which is beyond my comprehension.
In particular I have some fastq files which I'd like to trim with trim_galore for which reason I coded the following module:

process TRIM_GALORE {

    tag "$meta.id"

    input:
    tuple val(meta), path(reads)

    output:
    tuple val(meta), path("*_trimmed_val_*.fq.gz"), emit: reads
    path "*fastqc.{zip,html}",                      emit: fastqc
    path "*trimming_report.txt",                    emit: reports

    shell:
    read1       = reads[0]
    read2       = reads[1]
    lastPath    = read1.lastIndexOf(File.separator)
    read1Base   = read1.substring(lastPath+1)
    lastPath    = read2.lastIndexOf(File.separator)
    read2Base   = read2.substring(lastPath+1)

    '''
    trim_galore --paired \
                --quality 20 \
                --fastqc \
                --illumina \
                --gzip \
                --output_dir !{meta.id} \
                --basename !{meta.id}_trimmed \
                --cores !{task.cpus} \
                !{read1} \
                !{read2}

    mv !{read1Base}_trimming_report.txt !{meta.id}_trimmed_val_1.fq.gz_trimming_report.txt
    sed -i 's/Command line parameters:.*\$/Command line parameters: !{meta.id}_trimmed_val_1/g' !{meta.id}_trimmed_val_1.fq.gz_trimming_report.txt
    mv !{read2Base}_trimming_report.txt !{meta.id}_trimmed_val_2.fq.gz_trimming_report.txt
    sed -i 's/Command line parameters:.*\$/Command line parameters: ${meta.id}_trimmed_val_2/g' !{meta.id}_trimmed_val_2.fq.gz_trimming_report.txt
    '''
}

The corresponding workflow composition looks like something like this:

    // read QC
    TRIM_GALORE ( ch_cat_fastq )

    // splitting fastqs for HICUP parallelization
    SPLIT_FASTQ ( TRIM_GALORE.out.reads )

However, regardless of what I do I keep getting a No such variable: Exception evaluating property 'out' for nextflow.script.ChannelOut error. If I remove the out. I just get No such variable: reads

I couldn't find anything regarding this on the internet. Does anyone have a clue?

Daniel Malzl
@dmalzl
Nextflow version is 21.10.0
Christian Tischer
@tischi
Hi all, I am using nextflow to launch processes that rely on Java.
Processing them in parallel gives me Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
When executing them sequentially (executor.queueSize = 1) I don't get this error. Does anyone know how to fix this?
Karin Lagesen
@karinlag
hi all
I am wondering how I can specify options to a process outside of the process itself, that is, in a config file
specifically, I want to put the conda directive into a config file
one per process
note, I am still on dsl1
SowmyaPulapet
@SowmyaPulapet

Hi,
I am trying to run TransPi with the '--all' option, but the process halted abruptly stating the following error:

DEBUG nextflow.Session - Session aborted -- Cause: Operator into has been deprecated -- it's not available in DSL2 syntax
Jul-22 14:24:15.801 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.DeprecationException: Operator into has been deprecated -- it's not available in DSL2 syntax
I assume this is an issue with the new nextflow update. Is there a way to fix this?

jemma-nelson
@jemma-nelson
Hey all, you'll probably get better support on Slack than here:
https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html
BonaBeavis
@BonaBeavis
Hi, is it possible to debug the nextflow sscripts with a debugger?
santiagorevale
@santiagorevale
Hi there! How many possible combinations can be obtained for the "Run Name" between the two word sets? And how big is each set? Thanks!
Hussein Elgridly
@helgridly
Hi folks - quick Q. If I have a file in a versioned S3 bucket, can I refer to a non-latest version of it in Nextflow? e.g. s3://bucket/my.file?version=12345
jemma-nelson
@jemma-nelson
@BonaBeavis @santiagorevale @helgridly You may want to try the Slack community, this gitter really isn't active any more.
alexchole
@alexchole:matrix.org
[m]
"I'll help 10individuals how to earn $30,000 in 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message on Telegram by asking me (HOW) for more details on how to get started
https://t.me/+lD4Ec_gRjCljYjNk
Stijn van Dongen
@micans
It's been a while since I visited ... hello dear NF hackers! I'm puzzling over which Java is best to install on macos/M1 architecture (with the purpose of running NF of course). I see Oracle, there's brew cask, or even conda ... any recommendations/considerations?
Stijn van Dongen
@micans
I went with oracle aarch64 java, then hit an issue described in #2856, which was solved by the last comment's suggestion export JAVA_HOME=$(/usr/libexec/java_home -v 18)
Is this still a good place to hang around? Or is there a better place?
Moritz E. Beber
@Midnighter
Stijn van Dongen
@micans
:+1: :+1: :+1:
(I now see a few lines above Paolo posted the same :facepalm:
claraaudbtc
@claraaudbtc:matrix.org
[m]
Win up to $1000 in crypto trading when you invest with just the minimum of $100.
Signup and start investing your crypto with.
💎NO STRESS
💎NO REFERRAL NEEDED!!
💎NO REGISTRATION FEE!!
https://t.me/+fg0XTHR8CAo4Y2Fk
https://t.me/+fg0XTHR8CAo4Y2Fk
anoronh4
@anoronh4
i am having trouble understanding REVISION ID in the nextflow execution log (.nextflow/history). the documentation says it should be a git branch/tag/commit SHA, but i see only a long string that does not appear anywhere in my git log. is the documentation correct?
Anand Mayakonda
@PoisonAlien
Hello all. How do I sort the input files by reverse order i.e reverse-lexicographic-order ?
I have this command which lists all vcf files in lexicographic order SNV_vcf_files = file("03_snvs/*.hard-filtered.vcf.gz", sort: true) I want to sort them in reverse order.
Anand Mayakonda
@PoisonAlien
I just added reverse tag. SNV_vcf_files = file("03_snvs/*.hard-filtered.vcf.gz", sort: true).reverse() Do let me know if there is an elegant way of doing this.