Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 15:00
    bentsherman opened #3463
  • 14:44

    bentsherman on task-graph

    Add initial task graph [ci fast… (compare)

  • 11:56

    pditommaso on master

    Add sentence on bash options (#… (compare)

  • 11:52

    pditommaso on master

    Add sentence on bash options (#… (compare)

  • 11:52
    pditommaso closed #3454
  • 10:42
    pditommaso commented #3443
  • 09:57
    pditommaso synchronize #3443
  • 08:37
    pditommaso commented #3012
  • 07:17
    marcodelapierre commented #3012
  • 03:22
    aaronegolden commented #3443
  • 03:21
    aaronegolden commented #3443
  • 02:31
    aaronegolden synchronize #3443
  • Dec 01 22:00
    Jessime opened #3461
  • Dec 01 19:56
    bentsherman commented #3459
  • Dec 01 17:26
    corneliusroemer commented #3458
  • Dec 01 17:23
    corneliusroemer commented #3457
  • Dec 01 17:08
    bentsherman commented #3457
  • Dec 01 16:57
    bentsherman commented #3458
  • Dec 01 16:43
    bentsherman labeled #3457
  • Dec 01 16:43
    bentsherman labeled #3459
Asaf Peer
@asafpr
Thanks
ziltonvasconcelos
@ziltonvasconcelos

Hello all, I am facing an error trying to use nextflow with azure batch. Currently, I am using the nf-core/sarek pipeline and at the begining of the pipeline it gives me the error below:

Error executing process > 'MapReads (1543_18-1)'

Caused by:
Cannot find a VM for task 'MapReads (1543_18-1)' matching this requirements: type=Standard_E8_v3, cpus=16, mem=60 GB, location=brazilsouth

Any help will be appreciated,
Zilton.

4 replies
9d0cd7d2
@9d0cd7d2:matrix.org
[m]
I supose that somebody used Singularity on Nextflow. I'm trying to mount some local dirs on the container, but reading the documentation I faced this:
Nextflow expects that data paths are defined system wide, and your Singularity images need to be created having the mount paths defined in the container file system.
This mean that I need to create the Singularity image defining inside the mount points? Or I can "map" externally the dirs that I wan to mount inside? As the Docker style -v /something:/something
Sablin AMON
@wiztrust
Hello, I am new to Nextflow and would like to know how to send each task (process) via ssh for execution on a remote cluster with Nextflow? Thanks
ziltonvasconcelos
@ziltonvasconcelos

Hello people,
I could solve the previous error on Azure Batch and now another problem appear:

Caused by:
  At least one value of specified task container settings is invalid

Command executed:

  alleleCounter --version &> v_allelecount.txt 2>&1 || true
  bcftools --version &> v_bcftools.txt 2>&1 || true
  bwa version &> v_bwa.txt 2>&1 || true
  cnvkit.py version &> v_cnvkit.txt 2>&1 || true
  configManta.py --version &> v_manta.txt 2>&1 || true
  configureStrelkaGermlineWorkflow.py --version &> v_strelka.txt 2>&1 || true
  echo "2.7.1" &> v_pipeline.txt 2>&1 || true
  echo "21.10.6" &> v_nextflow.txt 2>&1 || true
  snpEff -version &> v_snpeff.txt 2>&1 || true
  fastqc --version &> v_fastqc.txt 2>&1 || true
  freebayes --version &> v_freebayes.txt 2>&1 || true
  freec &> v_controlfreec.txt 2>&1 || true
  gatk ApplyBQSR --help &> v_gatk.txt 2>&1 || true
  msisensor &> v_msisensor.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  qualimap --version &> v_qualimap.txt 2>&1 || true
  R --version &> v_r.txt 2>&1 || true
  R -e "library(ASCAT); help(package='ASCAT')" &> v_ascat.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  tiddit &> v_tiddit.txt 2>&1 || true
  trim_galore -v &> v_trim_galore.txt 2>&1 || true
  vcftools --version &> v_vcftools.txt 2>&1 || true
  vep --help &> v_vep.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  -

Command output:
  (empty)

Work dir:
  az://genomas-raros/work/11/f701b3eb21426f510b935f9e75da58

Any ideas?
Thanks in advance,
Zilton.

Paolo Di Tommaso
@pditommaso
What version of Nextflow are you using?
ziltonvasconcelos
@ziltonvasconcelos
version 21.10.6 build 5660
Paolo Di Tommaso
@pditommaso
I've had this same problem recently, however it's not clear what it's happening
please move this discussion here
shenkers
@shenkers

I'm using DSL2, and I've been building a workflow where I want to allow the user to use different "flavors" of the workflow by specifying command line parameters. I started by using the if()/else() blocks described in the documentation, but I found that when I have multiple configurable steps this pattern explodes into nested conditional logic that becomes hard to read.

With a little fiddling I found that I could define a variable that holds a reference to a runnable step, and then use this variable in the workflow block as though it was a regular process or workflow. I haven't seen this described in the documentation, but it was a concise way of describing what I wanted. Is there a more conventional (syntactic sugar) "nextflow-way" of doing this?

to give a more concrete example, the pattern I was using was:

process variant1 {
...
}

process variant2{
...
}

def variable_step = params.flag ? variant1.&run : variant2.&run

workflow {
  step1()
  variable_step()
  step3()
}

Do you think this is a "safe" thing to do? will it be stable-ish/compatible with future versions of nextflow?

shenkers
@shenkers

when I don't define that variable_step such that it points to the run method, or bind call() to run(), e.g.:

variable_step.metaClass.call = variable_step.&run

then I get this error:

Missing process or function with name 'call'

when a process-def is "called" in a workflow context is there something that does call()->run() binding at runtime? Why does assigning it to an intermediate variable change how the binding happens?

Paolo Di Tommaso
@pditommaso
hacking internal structures is not guaranteed to work. it's not enough an if else to eachive the same?
kaitlinchaung
@kaitlinchaung
Does anyone know what to do if a job finishes with Exit Code 0 but is still marked as 0? the .command.err and .command.out files are empty, and the job indeed finished successfully
Alex Mestiashvili
@mestia
How can I print a message if a channel is empty? I've tried something like that, but it prints the channel content when it is not empty files_ch.ifEmpty("emtpy").view() Also is there a way to exit the workflow if a channel is empty?
Tim Dudgeon
@tdudgeon

I'm having a problem defining optional outputs in DSL2. I have a workflow where part of it is optional:

main:
process1()
if (params.run_extra) {
  process2()
}

That works fine. But now I want to emit the outputs. I've tried:

emit:
process1.out
if (params.run_extra) {
  process2.out
}

But that doesn't seem to be allowed.

Any ideas?

Paolo Di Tommaso
@pditommaso
you need a conditional expression
1 reply
emit:
some_ch = params.run_extra ? process2.out : some_default
Jong Ha Shin
@JjongX
JjongX.png
jerovign
@jerovign
Hello, I was wondering if there is a way to ask nextflow to resubmit the run again and again. Because sometimes the job crashed due to incompatibility with the system, but it is random error. Running again the job with -resume function solve the problem.
3 replies
chbk
@chbk
#2546 was closed so I guess that functionality is not supported with DSL2 anymore. As an alternative I am attempting to execute Groovy code in a Closure provided to the subscribe method. Is it possible to catch an error that is thrown at that point? Basically something like this:
Channel.of(1, 2, 3).toList().subscribe({ list ->
  // Raise error
  throw new Exception("Something in the list is not valid")
})
// Would like to catch the error and stop execution here
println("Execution continues anyway") // This shouldn't be printed but it is
9d0cd7d2
@9d0cd7d2:matrix.org
[m]
Hi, I'm trying to bind one directory form the localhost to a Singularity container. I tried both containerOptions --volume /data/db:/db on the process section and also singularity.runOptions = '-B /data/db:/db' but I cannot mout it properly
I need to modify something on the Singularity configuration to allow this?
singularity.autoMounts = true is defined on my profile section too
I'm trying something like this, just for test:
echo true
containerOptions '--volume /tmp:/tmp/tmp-mounted'

script:
"""
cat /etc/*release >> /tmp/tmp-mounted/testing-bind.txt
echo "Hello world! From Singularity container" >> /tmp/tmp-mounted/testing-bind.txt
touch /tmp/tmp-mounted/thepipelinehasrun
ls /tmp/tmp-mounted/
cat /tmp/tmp-mounted/testing-bind.txt
"""

}
with a profile:
profiles { singularity { singularity.enabled = true singularity.autoMounts = true process.container = 'alpine.3.8.simg' } }
1 reply
ChillyMomo
@ChillyMomo709

Hi all,

It seems I still have 'configuration conflict' when I run awsbatch. Like following bug: nextflow-io/nextflow#2370 .

Configuration conflict
This value was submitted using containerOverrides.memory which has been deprecated and was not used as an override. Instead, the MEMORY value found in the job definition’s resourceRequirements key was used instead. More information about the deprecated key can be found in the AWS Batch API documentation.

Nextflow version

  Version: 21.10.6 build 5660
  Created: 21-12-2021 16:55 UTC 
  System: Linux 5.11.0-1022-aws
  Runtime: Groovy 3.0.9 on OpenJDK 64-Bit Server VM 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
  Encoding: UTF-8 (UTF-8)

How to solve this?

ChillyMomo
@ChillyMomo709
Issue does not seem to happen with NXF_VER=21.04.1 nextflow run main.nf
Jeffrey Massung
@massung
Is there a process directive I can use to fail a workflow? Maybe I can just throw an exception , but not sure if there's something nicer I should do instead? I basically have an if block in the directives section of a process, and if it's false I want to stop running and fail.
Nathan Spix
@njspix

I have a workflow that involves splitting up files per chromosome and then merging them later. To make the workflow a bit more flexible, I first pull the chromosomes out of the reference file using a bit of grep, so I have a channel with all the chromosome names. I can then do something like this:

input:
tuple val(id), path(file) from channel_a
each chr from chromosomes

output:
tuple val(id), val(chr), path(outfile) into channel_b

then I can group things up:

channel_b.groupTuple(by: 0)

and use that as input for the next process.
My question is, since the number of chromosomes is constant for any given run of the workflow, can I extract that value (e.g. map{ it.readLines().size() }) and feed that into groupTuple? I thought perhaps I could assign this value to a variable and then pass that variable to the groupTuple call but this doesn't work (the type of the variable is something fancy, not an Int).

1 reply
Shellfishgene
@Shellfishgene
The docs say that the merge operator will be removed soon. What's the replacement?
9d0cd7d2
@9d0cd7d2:matrix.org
[m]

Hi, this config worked for me :

singularity.enabled = true singularity.runOptions = "--bind /path:/path"

many thanks for your suggerence @tomraulet , finally Is working for me to!

xmzhuo
@xmzhuo

can I generate a val ouput emit from a script inside a process?
I tried to emit as val

process test {
output:
   val  val_var , emit: val_var
  shell:
"""
val_var=test
"""
}

Error:
Caused by:
Missing value declared as output parameter: val_var

I also tried to emit as env

process test{
output:
   env  val_var , emit: val_var
  shell:
"""
val_var=test
"""
}

When I tried to use it in the downstream process by calling test.out.val_var
Caused by:
No such property: val_var for class: ScriptC3863517AF925202A24F63BCD0003707

2 replies
Stathis
@isthisthat

hello, I'm just starting out in nextflow and seeking some strategic advice. What I'm trying to achieve is to run a workflow for a list of input samples. I can setup a workflow for a single sample, but how do I push multiple samples through it? (most examples in the docs show a single process) Here's where I'm stuck:

params.samples = ["samples/a","samples/b","samples/c"]
process step1 {
    input:
    file sample from samples_ch
    output:
    file 'result.bam' into step1_ch
    ...
}
process step2 {
    input:
    file bam from step1_ch
    output:
    file 'result.vcf' into step2_ch
    ...
}

This runs for sample a but not the rest, I'm suspecting because step2 only accepts one thing from step1_ch?
I can see two general strategies, either make a workflow for a single sample and then import that into a multi-sample wrapper, or enable each process to accept multiple inputs? Any advice would be greatly appreciated! Thanks

1 reply
Håkon Kaspersen
@hkaspersen
Hello everyone, I am converting my pipeline to use singularity images. I use images from biocontainers. I am trying to make my pipeline portable, and for now I have created a script that download these images to a user-specified directory to run from. However, some containers, such as R with specific packages, I could not find another solution than to build the image myself using a .def file with singularity build. This proved to work, but I am unsure how to make this portable. What are best practices regarding images, and how should i do this to optimize functionality and user-friendlyness?
9d0cd7d2
@9d0cd7d2:matrix.org
[m]
Hi, I tried to join nf-tower gitter channel, do somebody know if it's only for Enterprise users? Thanks in advance
maxulysse
@maxulysse:matrix.org
[m]
It's for everyone
9d0cd7d2
@9d0cd7d2:matrix.org
[m]
I cannot join the channel from the url, maybe need that somebody from the team puts me inside
Laurent Modolo
@l-modolo:matrix.org
[m]
Hi, I am trying unsuccessfully to implement the feedback loop pattern in DSL2. Is it possible to implement ?
4 replies
ebioman
@ebioman

if I see that correctly, the error strategy "ignore" will lead to "workflow.success"=true at the end.
I would like if running many samples to indeed ignore one failing one, but then check at the very end if any of them failed.
Is there a trace/object which can be accessed in main.nf where one could verify that at the end ? Something like

Sample success
1 true
1 true
1 true
1 true
1 false
1 true
1 true

This would allow to clean e.g. published data which would be otherwise orphan files

ebioman
@ebioman
Sorry cant edit, but obviously is should be different sample, doh
tkwitsil
@tkwitsil

I am having an issue where an imported module has an implicit workflow.onComplete handler. When I run the main workflow, the imported workflow.onComplete handler is being triggered, I assume because wf2's "workflow" is in the wf1 namespace. Example code:

# wf1.nf
include { subworkflow } from "./wf2"

// wf1 implicit workflow
workflow {
    main:
        println('wf1 implicit workflow called')
}

// pulls wf2 implicit workflow.onComplete into this namespace and executes 

----------------------------------------
#wf2.nf
//explicitly named workflow that is imported to wf1
workflow subworkflow {
    main:
        println('wf2 as subworkflow called')
}

// wf2 implicit workflow
workflow {
    main:
        println('wf2 implicit workflow called')
}

// wf2 implicit workflow.onComplete handler
workflow.onComplete {
    log.info('wf2 implicit workflow completed')
}

Command and output is:

$ nextflow run wf1.nf 
N E X T F L O W  ~  version 21.10.2
Launching `wf1.nf` [awesome_euclid] - revision: e050c16fcb
wf1 implicit workflow called
wf2 implicit workflow completed

Is there a way to avoid this namespace clash while keeping the workflow.onComplete handler for wf2? Or do I need to pull out the subworkflow in example above to it's own separate file and have wf1 import directly from that?

pouya ahmadvand
@pouya1991
Hi I am trying to use Nextflow to run singularity containers of my experiment pipeline. I am using the slurm as the executor. The problem I am facing is the the singularity has not been added to the node path and I then the Nextflow is not able to find the singularity and give this error:
env: ‘singularity’: No such file or directory
I submit my jobs through the head node which is different from the execution node.
I am wondering is there any way to append a custom path for singularity command look up?
Thanks
Peter Evans
@peterkevans
Hi all, first time poster... Sorry if this question has been asked but I'm struggling to find an answer.
I have a process that creates a bunch of fastq files and need to create a channel from them in order to scatter the next process. I'm believe I need something like the following output statement but I'm having trouble working out how to set sampleId to * i.e. the to get the sampleId from the file name. Any help would be greatly appreciated. (I'm using DSL2)
  output: 
  tuple val(sampleId), path("fastq_files/*_R{1,2}_001.fastq.gz"), emit: fastq
3 replies
awgymer
@awgymer

I have a situation where I need some dynamic input values for a process which I will need to fetch from a datastore as part of the pipeline. I was wondering what the best/accepted way of getting these values available to the process as variables is? My initial thought is to have the script that grabs the values from the datastore output a JSON file and then use a JSON reader in the process that requires them to access them?

Something like:

proc1 {
     output: 
        path patient_data.json
     script:
     """
     python get_patient_data.py
     """
} 

proc2 {
     input:
         path patient_data_file
         path other_file 
    output:
         path some_output.file
    script:
    patient_data = jsonSlurper.parse(patient_data_file)
    """
    the_command --opt1 ${patient_data['val1']} --opt2 ${patient_data['val2']} other_file
    """
}

Is this a reasonable solution? (I am aware the actual code above won't work because I haven't properly created the jsonslurper)