Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 05:54
    knservis edited #2473
  • 05:52
    knservis edited #2473
  • 05:52
    knservis opened #2473
  • 04:10
    manuelesimi edited #2471
  • 04:10
    manuelesimi edited #2471
  • 04:08
    manuelesimi labeled #2471
  • 04:03
    manuelesimi synchronize #2471
  • 04:03

    manuelesimi on container_properties

    Adjust format. Signed-off-by: … (compare)

  • 01:29
    Sid-Kr1 edited #2472
  • 01:24
    Sid-Kr1 opened #2472
  • Dec 01 19:38
    manuelesimi review_requested #2471
  • Dec 01 19:31
    manuelesimi opened #2471
  • Dec 01 19:14

    manuelesimi on container_properties

    Map the container options of th… (compare)

  • Dec 01 18:37
    mjhipp synchronize #2432
  • Dec 01 17:10
    ewels commented #2467
  • Dec 01 16:41
    pditommaso commented #2467
  • Dec 01 11:25
    stale[bot] closed #1766
  • Dec 01 11:25
    stale[bot] closed #2084
  • Dec 01 08:38
    tamasgal edited #2470
  • Dec 01 08:37
    tamasgal opened #2470
gianmore
@gianmore
Also how can I specify only error 140 and 143 and not all the other in the middle? I tried to separate them with a comma, but it didn't work
gpalou4
@gpalou4

Hi guys.
I have a simple doubt (I think). I just need to save my output process in an external directory so I'm using the publishDir function for it. However I need 2 things: i) use variables (collected from input tuple); ii) Create the directory because it doesn't exist.
I successfully run it like this:

    publishDir "${params.out_dir}/${tcga_project}/${tcga_barcode}/RSEM_quantification", mode: 'move'

    input:
    file STAR_bam_file from STAR_alignment_bam
    set val(sample_UUID), val(tcga_barcode), val(tcga_project) from samples_ch_3

But It doesn't create the directory. So I tried switching to this:

    outDir = file("${params.out_dir}/${tcga_project}/${tcga_barcode}/RSEM_quantification")
    outDir.mkdirs()
    publishDir outDir, mode: 'move'

    input:
    file STAR_bam_file from STAR_alignment_bam
    set val(sample_UUID), val(tcga_barcode), val(tcga_project) from samples_ch_3

However it states the error of:
No such variable: tcga_project

Any help in this situtation? Thanks :)

2 replies
hukai916
@hukai916
Hi all,
If exit 1, "ERROR MSG" is from a sub-workflow, the message will not be printed out to console, though it is inside .nextflow.log. I must use NXF_ANSI_LOG=false nextflow run main.nf to force the exit message to console.
Can anyone let me know why and how to solve it?
Regards,
nickhsmith
@nickhsmith
Hi all! I am trying to find how to set a profile in a config file. For example, I'm using a nf-core package, and am sick of running -profile docker at the command line level, and instead I would like to have in my nextflow.config file profile = docker such that it successfully imports all of the config options specified by the docker profile. thanks
3 replies
Brandon Cazander
@brandoncazander
I'm using collectFile in my workflow to combine the results of several upstream processes, and setting storeDir. However, when I resume my workflow (using AWS Batch executor), even though the upstream processes are correctly cached, the downstream process that consumes the output of collectFile always re-executes. I've tried sorting the input to collectFile with no luck.
4 replies
9d0cd7d2
@9d0cd7d2:matrix.org
[m]
Do you think if it's possible to decouple the executor destination from the user pipeline? My purpose is to have "one architecture pipeline" deciding the executor based on different metrics for each step of another pipeline ("user workflow"). I'm thinking on generate dynamically the user pipeline file, but I don't know if is possible to achieve this as a built-in functionality on the tool. Thanks in advance.
3 replies
xmzhuo
@xmzhuo
How to use docker from a private repo (user and password needed), can not find this in Scope docker:
 docker {
    enabled = true
}
process {
   withName:toy {
        container = 'user.abc.io/test/example'
   }
}
7 replies
Young
@erinyoung

How come

Channel
  .fromFilePairs( "${params.reads}/*{1,2}*.fastq.gz", size: 2 )
  .into { paired_reads }

does not work when the file starts with a number (like 7126-MS-1_100_S1_L005_R1_001.fastq.gz and 7126-MS-1_100_S1_L005_R2_001.fastq.gz or 21VR067049-211104_S1_L001_R1_001.fastq.gz and
21VR067049-211104_S1_L001_R2_001.fastq.gz

8 replies
Moritz E. Beber
@Midnighter
What is the recommended way to write a CSV file from nextflow? I can't seem to import the CSVPrinter https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVPrinter.html
David Mas-Ponte
@davidmasp
Hello all, I am using .collate() in my pipeline and then an implementation of this which works elegantly. My problem is with the -resume option. Some processes get cached, but the rest are re-scheduled. I am assuming the order changes from run to run before the collate operator, and thus assumes input has changed. Could this be it?
Also, what is the “canonical” way of sorting a channel? Maybe this?
Channel.from(2,1,3,4).toSortedList().fromList()
Brandon Cazander
@brandoncazander
When running with the AWS Batch executor, is it the main nextflow process that publishes files (using publishDir directive, or do the individual processes do this themselves?
5 replies
chittablog
@kumarusa2020

Hello Group members, I am trying a sample nextflow script using Azure Blob storage for work directory. I am getting an error and couldn't understand if my setup has anything to do, appreciate your response. Here is the error when trying to run my sample script connecting to Azure Blob storage

nextflow run tutorial.nf -w az://nextflow-work/work

image.png
3 replies
xmzhuo
@xmzhuo

How to deal with the empty path channel in process (dsl2)? I saw a solution for empty channel of value (https://github.com/nextflow-io/patterns/blob/master/docs/process-when-empty.adoc)

nextflow.enable.dsl=2
params.a = '*.txt'
params.b = ''
params.c = ''

process test {
   tag "test "
   echo true

   input:
   path file1
   path file2
   val  value1

   """
   echo $file1
   echo $file2
   echo $value1
   """

}

workflow {
    a_ch = Channel.fromPath(params.a)
    b_ch = Channel.fromPath(params.b)
    test(a_ch,b_ch,params.c)
}

The Error message:

N E X T F L O W  ~  version 21.04.3
Launching `code/test2.nf` [modest_colden] - revision: b4078eef6a
Missing `fromPath` parameter
rpetit3
@rpetit3:matrix.org
[m]
could try a_ch = params.a ? Channel.fromPath(params.a) : [] maybe?
1 reply
yedidya-be
@yedidya-be:matrix.org
[m]

Hi,

I'm trying to use Docker in Nextflow script and I'm getting the following error:

"connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Fr
un%2Fdocker.sock/v1.24/containers/create?name=nxf-8IoBF1Fj7HBmfAtOGRGpW5Fp": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'."

Does anyone have any advice on how to handle this?

yedidya-be
@yedidya-be:matrix.org
[m]

:point_up: Edit: Hi,

I'm trying to use Docker in Nextflow script and I'm getting the following error:

"connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Fr
un%2Fdocker.sock/v1.24/containers/create?name=nxf-8IoBF1Fj7HBmfAtOGRGpW5Fp": dial unix /var/run/docker.sock: connect: permission denied.
 See 'docker run --help'."

Does anyone have any advice on how to handle this?

2 replies
Alan Hoyle
@alanhoyle
In a profile in the Nextflow config, it sets process.errorStrategy = 'finish'. Can someone remind me how to override that at the command line? I want to set it to 'retry' (and I assume I can do something similar to set maxTries = 4)
1 reply
Nicolas Mendoza
@nmendozam

Hello,

I am trying to run nextflow inside a docker and from there: run other dockers in the host machine. For that I have built a docker image with nextflow and docker inside, as so:

FROM nextflow/nextflow

# Adds ability to run docker in the host
RUN amazon-linux-extras install docker -y

WORKDIR /nextflow

But when running the below line, it says Command error: /bin/bash: .command.sh: No such file or directory

docker run -v /var/run/docker.sock:/var/run/docker.sock custom/nextflow bash -c 'nextflow run hello -with-docker nextflow/examples'

Does anyone have an idea on how to fix this?

Richard Corbett
@RichardCorbett
Hi all, is there a way to "flush" the publish command? I understand it is asynchronous by default, but I have a pipeline that runs for 7-10 days and some of the useful results are complete after the first day. Would be good if I could request the results from day 1 are "published" soon after completion.
saphir746
@saphir746
Hello Everyone. Running NF as part of a custom made script, wrapping around MatLaB commands. I have a problem where some of my sub-processes fail, but I can't workout why: Caused by:
Process init_Config (3) terminated for an unknown reason -- Likely it has been terminated by the
external system
Looking at the log files of the other 'init_Config' processes, nothing differentiates them. No error flags around wall-time or ressource usage. How do you trouble-shoot that?
Scott Yourstone
@islandhopper81

Hello everyone,

Is it possible to update process information using withLabel on the command line? So something like:

nextflow run hello_world.nf -process.withLabel:hello_world.cpus 2

I know that something as simple as this works:

nextflow run hello_world.nf -process.cpus 2

but I want to target a particular process.

Håkon Kaspersen
@hkaspersen

hello everyone, I am using singularity images from biocontainers for most of my processes, where they are pre-downloaded and used directly (as the cluster nodes don't have internet access). However, I am having issues with the Prokka image from that site. I pull the images from biocontainers with singularity pull, and use that image directly in the nextflow pipeline. However, it seems that nextflow doesn't want to use the prokka image directly, it rather wants to download it from docker. Note that it works fine for all other processes that have been downloaded the same way. Does anyone know anything about this image? https://biocontainers.pro/tools/prokka
(Note: I have tried several versions of the singularity image, all give the same result).
My nextflow.config:

params.container_dir    = 'path/to/images'

singularity {
        enabled         = true
        autoMounts      = true
}

The container is called like this in the prokka process:

process PROKKA {
        container "${container_dir}/prokka:1.14.5--pl526_1"

Thanks in advance!

mbahin
@mbahin
Hi, I have a (stupid?) question :). What is the synthax for comments in the script block?
2 replies
Tj-Idowu
@Tj-Idowu

Hello,

I am trying to figure out how to separate and categorize some files based on some string values in their name. For example, if the files are named;
X_filename_f0_r1-1_1_R1.fastq
X_filename_f0_r0-1_1_R1.fastq
X_filename_f0_r0-1_2_R1.fastq
X_filename_f0_r1-0_3_R1.fastq

X_filename_f1_r1-1_1_R1.fastq
X_filename_f1_r0-1_1_R1.fastq
X_filename_f1_r0-1_2_R1.fastq
X_filename_f1_r1-0_3_R1.fastq

X_filename_f2_r1-1_1_R1.fastq
X_filename_f2_r0-1_1_R1.fastq
X_filename_f2_r0-1_2_R1.fastq
X_filename_f2_r1-0_3_R1.fastq

X_filename_f3_r1-1_1_R1.fastq
X_filename_f3_r0-1_1_R1.fastq
X_filename_f3_r0-1_2_R1.fastq
X_filename_f3_r1-0_3_R1.fastq

How can I separate the files based on f0, f1, f2, f3 and have each category saved under a different name? Then I want to separate the same files based on the numbers after the "r" but before the underscore. There are a lot of different numbers so I can't be specific in the script about the numbers to select. These same files will then be separated again based on the number before "_R1". So in the end, I will have a category of files based on f-number then a subcategory based on r number then another subcategory under that based on a number.

Thanks
Tj

Pablo Riesgo-Ferreiro
@priesgo
hi all,
I have searched for error handling in Nextflow docs without success. Can someone point me in the right direction?
I just want to do some sanity checks on the inputs of a process and raise a controlled error if certain error conditions are met.
Thanks!
Groovy's assert did not seemed to work ...
forget it, Groovy's assert methods do work!
rubber duck effect
ChillyMomo
@ChillyMomo709

Hi all,

My challenge of my code is to create input for pairwise comparison. Based on the comparison list, I would like to select the samples that are in that group. I have tried the following but not with complete success as I receive a DataflowBroadcast around DataflowStream[?]

bams = Channel.from(["sampleX_1", "groupX", "bamX_1", "baiX_1"], 
                            ["sampleX_2", "groupX", "bamX_2", "baiX_2"], 
                            ["sampleY_1", "groupY", "bamY_1", "baiY_1"], 
                            ["sampleY_2", "groupY", "bamY_2", "baiY_2"],
                            ["sampleZ_1", "groupZ", "bamZ_1", "baiZ_1"], 
                            ["sampleZ_2", "groupZ", "bamZ_2", "baiZ_2"])

 comparison_list = Channel.from(["groupX", "groupY"],["groupX", "groupZ"])

 group_input = comparison_list_file.map{ it ->
                                                    def bam_by_group = bam.groupTuple(by:1)
                                                    def compare1 = it[0]
                                                    def compare2 = it[1]
                                                    def group_ctrl = bam_by_group.first{ it[1] == compare1}
                                                    def group_case = bam_by_group.first{ it[1] == compare2}
                                                    def group_input = group_ctrl.combine(group_case)

                                                    return group_input.view() // - desired outcome, see below
                                                    }.view() // results into - DataflowBroadcast around DataflowStream[?]

I like the result to have it in this format:

        // [[sampleX_1, sampleX_2], groupX, [bamX_1, bamX_2], [baiX_1, baiX_2], [sampleY_1, sampleY_2], groupY, [bamY_1, bamY_2], [baiY_1, baiY_2]]
        // [[sampleX_1, sampleX_2], groupX, [bamX_1, bamX_2], [baiX_1, baiX_2], [sampleZ_1, sampleZ_2], groupZ, [bamZ_1, bamZ_2], [baiZ_1, baiZ_2]]

Of which group_input.view() does but how can I use that?

Thanks!

Christopher Hartl
@chartl-rancho

On a related note, I'm trying to group process outputs together for downstream paired processing. I have a digest file that looks like

sample_id,read1,read2,control_id
cell1_ctl,A.fq.gz,B.fq.gz,
cell1_trt1,C.fq.gz,D.fq.gz,cell1_ctl
cell1_trt2,E.fq.gz,F.fq.gz,cell1_ctl
cell2_ctl,G.fq.gz,H.fg.gz,
cell2_trt1,I.fq.gz,J.fq.gz,cell2_ctl
cell2_trt2,K.fq.gz,L.fq.gz,cell2_ctl

and I have a basic pipeline that drives from per-column channels and looks like

aln_bams = bwamem(samples_ch, read1_ch, read2_ch)
filt_bams = mapq_filter(aln_bams, 30)

but now i need to get to bam channels that look like

[cell1_trt1.bam, cell1_trt2.bam, cell2_trt1.bam, cell2_trt2.bam]
[cell1_ctl.bam, cell1_ctl.bam, cell2_ctl.bam, cell2_ctl.bam]

the issue is I can't find a way to join the filtered bams with the original IDs. In DSL=1, I could merge; but using DSL=2 I need an index on which to join, and there's no such thing as .mapWithIndex.

Any ideas?

2 replies
Arijit Panda
@arpanda
Is there any example/usage available of publishDir in DSL2 ?
5 replies
Arijit Panda
@arpanda
Also, is it possible to set the publishDir in the configuration file?
Arijit Panda
@arpanda

is there any usage example for this. nextflow-io/nextflow#524

I did

process {
    cpus = 4
    withLabel:test2 {
        publish_dir = 'result_data'
    }
}

And got the following warning

WARN: Access to undefined parameter `publish_dir` -- Initialise it to a default value eg. `params.publish_dir = some_value`
WARN: Process `test2` publishDir path contains a variable with a null value
WARN: Process `test2` publishDir path contains a variable with a null value
WARN: Process `test2` publishDir path contains a variable with a null value
WARN: Process `test2` publishDir path contains a variable with a null value
WARN: Process `test2` publishDir path contains a variable with a null value
Cameron Reid
@CJREID
Within the 'workflow' on DSL2, is it possible to simply assert that one process does not begin until another is complete? My first process downloads databases that are required by later processes but cannot be practically used as channels to control the order of execution.
6 replies
ChillyMomo
@ChillyMomo709

Hi all,

My challenge of my code is to create input for pairwise comparison. Based on the comparison list, I would like to select the samples that are in that group. I have tried the following but not with complete success as I receive a DataflowBroadcast around DataflowStream[?]

bams = Channel.from(["sampleX_1", "groupX", "bamX_1", "baiX_1"], 
                            ["sampleX_2", "groupX", "bamX_2", "baiX_2"], 
                            ["sampleY_1", "groupY", "bamY_1", "baiY_1"], 
                            ["sampleY_2", "groupY", "bamY_2", "baiY_2"],
                            ["sampleZ_1", "groupZ", "bamZ_1", "baiZ_1"], 
                            ["sampleZ_2", "groupZ", "bamZ_2", "baiZ_2"])

 comparison_list = Channel.from(["groupX", "groupY"],["groupX", "groupZ"])

 group_input = comparison_list_file.map{ it ->
                                                    def bam_by_group = bam.groupTuple(by:1)
                                                    def compare1 = it[0]
                                                    def compare2 = it[1]
                                                    def group_ctrl = bam_by_group.first{ it[1] == compare1}
                                                    def group_case = bam_by_group.first{ it[1] == compare2}
                                                    def group_input = group_ctrl.combine(group_case)

                                                    return group_input.view() // - desired outcome, see below
                                                    }.view() // results into - DataflowBroadcast around DataflowStream[?]

I like the result to have it in this format:

        // [[sampleX_1, sampleX_2], groupX, [bamX_1, bamX_2], [baiX_1, baiX_2], [sampleY_1, sampleY_2], groupY, [bamY_1, bamY_2], [baiY_1, baiY_2]]
        // [[sampleX_1, sampleX_2], groupX, [bamX_1, bamX_2], [baiX_1, baiX_2], [sampleZ_1, sampleZ_2], groupZ, [bamZ_1, bamZ_2], [baiZ_1, baiZ_2]]

Of which group_input.view() does but how can I use that?

Thanks!

I modified map_join function created by multimeric (nextflow-io/nextflow#559). I extended to function to "loop over" the keys and it worked!

kobelavaerts
@kobelavaerts
Hi all,
Is there a way to specify a custom path for the .nextflow.log file in the nextflow.config file? I know it's possible on the command-line using the -log option but I would like to specify it in the nextflow.config so I can modify the path based on our params.outdir parameter.
Arijit Panda
@arpanda

How to specify the thread value in nextflow script for sge jobs?

For local jobs, cpu information is provided using the following configuration settings.

process {
    cpus = 12
}

And after that that value is used in $task.cpus variable. But in can of SGE executor, the cpus variable is not allowed. So I provided threads using -pe option. But unable to access that value in the script.
Here is my configuration setup.

process {
    executor = 'sge'
    clusterOptions = '-V -pe threaded 13'

}

So, can someone suggest a solution for this. Thanks in advance.

Christopher Hartl
@chartl-rancho

Is there a practical difference (in DSL2) between

output:
   val sample
  path file

and

output:
  tuple val(sample), path(file)

?

4 replies
limin
@limin321

hi, all. I am brand-new to nextflow. Just recently installed nextflow on my local computer and ran into the follow error. Here is my code,

num = Channel.from( 1, 2, 3 )

process test {
   input:
   val num

   script:
   println "echo process test $num"

}
`

Here is my error: N E X T F L O W ~ version 21.10.0
Launching process.nf [irreverent_feynman] - revision: 386318b60a
executor > local (3)
executor > local (3)
[b7/c314a2] process > test (1) [100%] 3 of 3, failed: 3 ✘
echo process test 2
echo process test 1
echo process test 3
Error executing process > 'test (3)'

Caused by:
Process test (3) terminated with an error exit status (127)

Command executed:

null

Command exit status:
127

Command output:
(empty)

Command error:
.command.sh: line 2: null: command not found

Work dir:
/Users/limin/limin_practice/nextflow/work/37/b1c3dd131fb6078750b0f16150fe40

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

Can anyone help me understand what does the error message mean? How should I correct my codes?

5 replies
Jemma Nelson
@fwip
I've got some Best-Practices questions about composing dynamic directives.
We're currently using the basic strategy described in the docs like memory { 2.GB * task.attempt }, with separate labels that use the same retry-with-increased-resources strategy (like low_mem and high_mem) to coarsely indicate the size of the job. The label definitions are specified in the configuration file, and the labels are applied in the process file.
Is there a way to keep the "strategy" in the config file (x.GB * task.attempt), while the actual amount (2.GB, etc) varies based on some calculation done by the process directive?
Or is the strategy something I should be writing a function for, and then applying to the closure that calculates the value? Like memory (retryFunc { inputfile.size * 3 })
Dennis Hendriksen
@dennishendriksen

after upgrading Nextflow 21.04.3 to 21.10.3 my workflow fails with the following error:

groovy.lang.MissingMethodException: No signature of method: nextflow.script.WorkflowDef.nr_records() is applicable for argument types: ([Ljava.lang.Object;) values: [[/gro
ups/solve-rd/tmp10/projects/vip/nextflow/git/vip/test/output/test_snv/.nxf_work/92/1158ff50d7175029ff592f11a241f3/snv_prepared.vcf.gz.stats]]
        at nextflow.script.WorkflowParamsResolver.invokeMethod(WorkflowDef.groovy:217)
def nr_records(statsFilePath) {
  ...
}
  channel.fromPath(params.input) \
      | prepare \
      | branch {
          small: nr_records(it.last()) <= params.chunk_size
          large: true
        }
      ...
process prepare {
  ...
  output:
    tuple ..., path("${vcfOutputPath}.stats")
  ...

I've tried updating in the caller and callee in various ways without success. does anyone have a clue what changed between Nextflow versions?

2 replies
Marcello Salvi
@m.salvi_gitlab
Dear Nextflow team, I have a question related with GCP usage. Is it possible to use the service account to crosstalk beetween different project? (aka I have my file in a bucket in a project and I want to use the VM GCP life sciences in an other one)
Wenjun Kang
@wkang0
How to change the Singularity PATH with SINGULARITYENV_PREPEND_PATH=/good/stuff/at/beginning in nextflow?
priyambial123
@priyambial123

Hello, I installed next flow in the computing cluster but I am not able to use the docker files required for running deep variant analysis. I used the following command: nextflow run nf-core/deepvariant -profile test,docker

N E X T F L O W ~ version 21.10.4
Launching nf-core/deepvariant [lethal_tesla] - revision: 2b5486356c [master]

Downloading plugin nf-amazon@1.3.4

                                      ,--./,-.
      ___     __   __   __   ___     /,-._.--~'
|\ | |__  __ /  ` /  \ |__) |__         }  {
| \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                      `._,._,'

nf-core/deepvariant v1.0"

Pipeline Name : nf-core/deepvariant
Pipeline Version: 1.0
Bam file : https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/NA12878_S1.chr20.10_10p1mb.bam
Bed file : https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/test_nist.b37_chr20_100kbp_at_10mb.bed
Reference genome: hg19chr20
Fasta Ref : s3://deepvariant-data/genomes/hg19chr20/chr20.fa
Fasta Index : s3://deepvariant-data/genomes/hg19chr20/chr20.fa.fai
Fasta gzipped : s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz
Fasta gzipped Index: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.fai
Fasta bgzip Index: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.gzi
Max Memory : 6 GB
Max CPUs : 2
Max Time : 2d
Model : wgs
Output dir : results
Working dir : /scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master/work
Container Engine: docker
Container : nfcore/deepvariant:1.0
Current home : /scicore/home/cichon/thirun0000
Current user : thirun0000
Current path : /scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master
Script dir : /scicore/home/cichon/thirun0000/.nextflow/assets/nf-core/deepvariant

Config Profile : test,docker

executor > local (1)
[- ] process > preprocess_bam -
[- ] process > make_examples -
[- ] process > call_variants -
[- ] process > postprocess_variants -
[e7/dec5cb] process > get_software_versions [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
Error executing process > 'get_software_versions'

Caused by:
Process get_software_versions terminated with an error exit status (127)

Command executed:

echo 1.0 &> v_nf_deepvariant.txt
echo 21.10.4 &> v_nextflow.txt
ls /opt/conda/pkgs/ &> v_deepvariant.txt
python --version &> v_python.txt
pip --version &> v_pip.txt
samtools --version &> v_samtools.txt
lbzip2 --version &> v_lbzip2.txt
bzip2 --version &> v_bzip2.txt
scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
127

Command output:
(empty)

Command error:
.command.run: line 279: docker: command not found

Work dir:
/scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master/work/e7/dec5cbd65b63e4fdd5ccced4f1e516

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

Fastagz file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz

Fai file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.fai

gzfai file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.fai

please specify --bed option (--bed bedfile)

https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/NA12878_S1.chr20.10_10p1mb.bam not found

gzi file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.gzi

tomSimonet
@tomSimonet

hello
is it possible to make iteration over processes in dsl2 ?, as in https://github.com/nextflow-io/patterns/blob/master/docs/feedback-loop.adoc
what I tried doesn't work
(to annotate vcf with several databases)

take:
    entrees_vcf // tuple [info1, info2, info3, vcf]
    bdds // collection of collections: [ [bdd1,index1,tag1,fields1], [bdd2,index2,tag2,fields2], ...]

main:
input_ch = input_vcf
            .concat( bdds.map{it[0]}, Channel.of(0) ).flatten().toList()
            .mix(
                Annotation_bdd_vcf.out
                    .concat( bdds )
                    .toList()
                    .map { ch_tuple, bdds ->
                        if ( ch_tuple[4] < bdds.size() ) { //(bdds[iteration]) {
                            return [ ch_tuple[0], ch_tuple[1], ch_tuple[2], ch_tuple[3], bdds[iteration], ch_tuple[4] ].flatten()
                        }
                    }
            )
Annotation_bdd_vcf( input_ch )

in Annotation_bdd_vcf input, the last variable (coming from Channel.of(0)) is incremented each time the process is executed

4 replies
xmzhuo
@xmzhuo
azurebatch after completetion, pools terminated succesfully, but the Jobs still active
I put this two in the azure batch config
deletePoolsOnCompletion = true
deleteJobsOnCompletion = true
Konstantinos Servis
@knservis
Hi folks. I have been reading through the issues in order to find a good way to configure nextflow for my use-case. It involves running all tasks using the slurm executor, however I need to run different tasks on different clusters. It seems like clusters in not supported (slurm commands -M option) and the only thing that is mentioned in the documentation and the issues is using env.SLURM_CLUSTERS. Unfortunately as I am reading in another issue (and I have confirmed by trying) that is not valid in a process { withLabel: xxx {cant put env here}} selector block. Is my only option a PR?
Konstantinos Servis
@knservis
I submitted an issue here nextflow-io/nextflow#2473 I hope it makes sense