nextflow-io/nextflow

Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html

Dennis Hendriksen

@dennishendriksen

after upgrading Nextflow 21.04.3 to 21.10.3 my workflow fails with the following error:

groovy.lang.MissingMethodException: No signature of method: nextflow.script.WorkflowDef.nr_records() is applicable for argument types: ([Ljava.lang.Object;) values: [[/gro
ups/solve-rd/tmp10/projects/vip/nextflow/git/vip/test/output/test_snv/.nxf_work/92/1158ff50d7175029ff592f11a241f3/snv_prepared.vcf.gz.stats]]
        at nextflow.script.WorkflowParamsResolver.invokeMethod(WorkflowDef.groovy:217)

def nr_records(statsFilePath) {
  ...
}

  channel.fromPath(params.input) \
      | prepare \
      | branch {
          small: nr_records(it.last()) <= params.chunk_size
          large: true
        }
      ...

process prepare {
  ...
  output:
    tuple ..., path("${vcfOutputPath}.stats")
  ...

I've tried updating in the caller and callee in various ways without success. does anyone have a clue what changed between Nextflow versions?

2 replies

Marcello Salvi

@m.salvi_gitlab

Dear Nextflow team, I have a question related with GCP usage. Is it possible to use the service account to crosstalk beetween different project? (aka I have my file in a bucket in a project and I want to use the VM GCP life sciences in an other one)

Ghost

@ghost~61847cca6da037398489d4e6

How to change the Singularity PATH with SINGULARITYENV_PREPEND_PATH=/good/stuff/at/beginning in nextflow?

priyambial123

@priyambial123

Hello, I installed next flow in the computing cluster but I am not able to use the docker files required for running deep variant analysis. I used the following command: nextflow run nf-core/deepvariant -profile test,docker

N E X T F L O W ~ version 21.10.4
Launching nf-core/deepvariant [lethal_tesla] - revision: 2b5486356c [master]

Downloading plugin nf-amazon@1.3.4

                                      ,--./,-.
      ___     __   __   __   ___     /,-._.--~'
|\ | |__  __ /  ` /  \ |__) |__         }  {
| \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                      `._,._,'

nf-core/deepvariant v1.0"

Pipeline Name : nf-core/deepvariant
Pipeline Version: 1.0
Bam file : https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/NA12878_S1.chr20.10_10p1mb.bam
Bed file : https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/test_nist.b37_chr20_100kbp_at_10mb.bed
Reference genome: hg19chr20
Fasta Ref : s3://deepvariant-data/genomes/hg19chr20/chr20.fa
Fasta Index : s3://deepvariant-data/genomes/hg19chr20/chr20.fa.fai
Fasta gzipped : s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz
Fasta gzipped Index: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.fai
Fasta bgzip Index: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.gzi
Max Memory : 6 GB
Max CPUs : 2
Max Time : 2d
Model : wgs
Output dir : results
Working dir : /scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master/work
Container Engine: docker
Container : nfcore/deepvariant:1.0
Current home : /scicore/home/cichon/thirun0000
Current user : thirun0000
Current path : /scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master
Script dir : /scicore/home/cichon/thirun0000/.nextflow/assets/nf-core/deepvariant

Config Profile : test,docker

executor > local (1)
[- ] process > preprocess_bam -
[- ] process > make_examples -
[- ] process > call_variants -
[- ] process > postprocess_variants -
[e7/dec5cb] process > get_software_versions [100%] 1 of 1, failed: 1 ✘
Execution cancelled -- Finishing pending tasks before exit
Error executing process > 'get_software_versions'

Caused by:
Process get_software_versions terminated with an error exit status (127)

Command executed:

echo 1.0 &> v_nf_deepvariant.txt
echo 21.10.4 &> v_nextflow.txt
ls /opt/conda/pkgs/ &> v_deepvariant.txt
python --version &> v_python.txt
pip --version &> v_pip.txt
samtools --version &> v_samtools.txt
lbzip2 --version &> v_lbzip2.txt
bzip2 --version &> v_bzip2.txt
scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
127

Command output:
(empty)

Command error:
.command.run: line 279: docker: command not found

Work dir:
/scicore/home/cichon/thirun0000/my-pipelines/nf-core/deepvariant-master/work/e7/dec5cbd65b63e4fdd5ccced4f1e516

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

Fastagz file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz

Fai file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.fai

gzfai file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.fai

please specify --bed option (--bed bedfile)

https://github.com/nf-core/test-datasets/raw/deepvariant/testdata/NA12878_S1.chr20.10_10p1mb.bam not found

gzi file not found: s3://deepvariant-data/genomes/hg19chr20/chr20.fa.gz.gzi

tomSimonet

@tomSimonet

hello
is it possible to make iteration over processes in dsl2 ?, as in https://github.com/nextflow-io/patterns/blob/master/docs/feedback-loop.adoc
what I tried doesn't work
(to annotate vcf with several databases)

take:
    entrees_vcf // tuple [info1, info2, info3, vcf]
    bdds // collection of collections: [ [bdd1,index1,tag1,fields1], [bdd2,index2,tag2,fields2], ...]

main:
input_ch = input_vcf
            .concat( bdds.map{it[0]}, Channel.of(0) ).flatten().toList()
            .mix(
                Annotation_bdd_vcf.out
                    .concat( bdds )
                    .toList()
                    .map { ch_tuple, bdds ->
                        if ( ch_tuple[4] < bdds.size() ) { //(bdds[iteration]) {
                            return [ ch_tuple[0], ch_tuple[1], ch_tuple[2], ch_tuple[3], bdds[iteration], ch_tuple[4] ].flatten()
                        }
                    }
            )
Annotation_bdd_vcf( input_ch )

in Annotation_bdd_vcf input, the last variable (coming from Channel.of(0)) is incremented each time the process is executed

4 replies

xmzhuo

@xmzhuo

azurebatch after completetion, pools terminated succesfully, but the Jobs still active
I put this two in the azure batch config

deletePoolsOnCompletion = true
deleteJobsOnCompletion = true

Konstantinos Servis

@knservis

Hi folks. I have been reading through the issues in order to find a good way to configure nextflow for my use-case. It involves running all tasks using the slurm executor, however I need to run different tasks on different clusters. It seems like clusters in not supported (slurm commands -M option) and the only thing that is mentioned in the documentation and the issues is using env.SLURM_CLUSTERS. Unfortunately as I am reading in another issue (and I have confirmed by trying) that is not valid in a process { withLabel: xxx {cant put env here}} selector block. Is my only option a PR?

Konstantinos Servis

@knservis

I submitted an issue here nextflow-io/nextflow#2473 I hope it makes sense

Pierre Lindenbaum

@lindenb

hi all, in DSL2 is it possible to use a publishDir directive using a variable from input: ? I tried

publishDir meta.publishDir, mode: 'copy', saveAs: {filename -> meta.publishDir?filename:null}

input:
        tuple val(meta),val(L)
(...)

but got

No such variable: meta

1 reply

Pierre Lindenbaum

@lindenb

@Midnighter that worked ! thanks ! :-)

1 reply

Luca Cozzuto

@lucacozzuto

Dear all, I got this error when running my pipeline in AWS and resuming

and for this the resume is not working

have you faced something similar?

nextflow.file.FileHelper - Can't check if speci
fied path is NFS (1): /mop2-bucket-1/scratch

Luca Cozzuto

@lucacozzuto

@pditommaso no help? :(

1 reply

Tim Dudgeon

@tdudgeon

I'm not clear on the use of publishDir with DSL2. This announcement suggested that there would be improvements here, and I found this that seems to provide some mechanism, but I was assuming that you should be able to define a process that does not need to worry about publishDir and let the workflow choose which outputs to publish? Is this possible? Any examples?

Steven P. Vensko II

@spvensko_gitlab

Is it possible to use a .count() channel as the size parameter for GroupTuples?

I currently have:

bcftools_index_somatic.out.vcfs_w_csis.groupTuple(by: [0, 1, 2, 3], size: extract_chroms_from_bed.out.chroms_list.count()).set{ vcfs_by_patient }

But I get the error:

Value 'DataflowVariable(value=null)' cannot be used in in parameter 'size' for operator 'groupTuple' -- Value don't match: class java.lang.Integer

Is it possible to convert the .count() channel into something consumable by size:?

1 reply

KyleStiers

@KyleStiers

Does anybody know of resources with more in-depth details on implementation patterns of the sql plugin (https://github.com/nextflow-io/nf-sqldb). I've got it working for basic functionality, but it would be very useful to expand on the documentation a bit or add some examples of how to push/pull to/from a database within a process for example.

3 replies

Alexander Toenges

@atpoint:matrix.org

[m]

I remember there was an inbuilt nf way to display all params in a tabular output. Not $params because this gives a comma-separates map, but something that returns a table. Any ideas?

Haruka Ozaki

@yuifu

Hi! I sometimes get a "file not found" error when using Channel.fromFilePairs with URLs.
Is there any way to set a timeout duration?

Vlad Kiselev

@wikiselev

What is a common way of making a process optional? (e.g. the process execution depends on a boolean parameter). So far I've only come up with an idea of using an empty channel for this. Any other ideas?

And of course I found it after posting: https://nextflow-io.github.io/patterns/index.html#_solution_18

Reed Bell

@utlandr

Does Nextflow support pulling images from a private register for the google-lifesciences executor? I keep getting a pull access denied errorhere (running docker pull separately works fine).

Reed Bell

@utlandr

Update: Totally supported. It appears you need to have docker.fixOwnership = true (and install procps in your image otherwise Nextflow will complain that you don't have ps installed). Best guess is that the mismatch in the ids for the owner of the image manifest file was preventing it from being accessed. Haven't tried it but a quick peek at the source code suggests that setting NXF_OWNER will also make things work.

2 replies

Dennis Hendriksen

@dennishendriksen

We are running a Nextflow workflow with Singularity containers and the SLURM executor. Conceptually I am struggling:

the overhead for starting and stopping a job in SLURM is very high

I can only use one container per process / looking at nf-core modules Nextflow seems to prefer small processes

Dennis Hendriksen

@dennishendriksen

In my use case I have a elegant workflow with parallelization that in practice performs horribly.

3 replies

I was wondering if anyone else recognizes this struggle. How do you deal with this issue?

ericart89

@ericart89

Hello, anyone knows if it is possible to apply "resume" only for certain processes instead of all of them??

1 reply

Moritz E. Beber

@Midnighter

Does anyone know if nextflow uses log4j? There is a security vulnerability https://www.lunasec.io/docs/blog/log4j-zero-day/

9d0cd7d2

@9d0cd7d2:matrix.org

[m]

hi! Im trying to run some processes with Singularity containers, is it possible to bind internal container directories to the outside dirs on the host?

Brandon Cazander

@brandoncazander

Is there a way to specify a timeout for how long processes stay in a pending state? For example, I want to optimistically use one queue that has unused capacity, but if jobs are pending for x minutes, I would like them to fail and retry on another queue.

Paolo Di Tommaso

@pditommaso

@Midnighter nextflow uses logback

4 replies

Jeremy Leipzig

@leipzig

I'm having trouble with google life sciences.

Command error:
  Execution failed: generic::failed_precondition: while running "nf-2f69f94540149df9eda94c49022f51cc-main": unexpected exit status 1 was not ignored

the bucket is being used as a work directory but there is something not producing .command.out .command.err ..command.trace

i'm using nextflow-21.10.5 - any workflow does this including hello-world

Jeremy Leipzig

@leipzig

one clue might be

gsutil cat gs://mygcpbucket/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/google/logs/output
/bin/bash: /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log: Permission denied
+ trap 'err=$?; exec 1>&2; gsutil -m -q cp -R /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log || true; [[ $err -gt 0 || $GOOGLE_LAST_EXIT_STATUS -gt 0 || $NXF_DEBUG -gt 0 ]] && { ls -lah /nextflow/f4/72c513e7a923fb0c80b30fc74c669d || true; gsutil -m -q cp -R /google/ gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d; } || rm -rf /nextflow/f4/72c513e7a923fb0c80b30fc74c669d; exit $err' EXIT
+ err=1
+ exec
+ gsutil -m -q cp -R /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log
+ [[ 1 -gt 0 ]]
+ ls -lah /nextflow/f4/72c513e7a923fb0c80b30fc74c669d
total 40K
drwxr-xr-x 3 root root 4.0K Dec 10 23:51 .
drwxr-xr-x 3 root root 4.0K Dec 10 23:51 ..
-rw-r--r-- 1 root root 3.3K Dec 10 23:51 .command.log
-rw-r--r-- 1 root root 5.3K Dec 10 23:51 .command.run
-rw-r--r-- 1 root root   36 Dec 10 23:51 .command.sh
drwx------ 2 root root  16K Dec 10 23:50 lost+found
+ gsutil -m -q cp -R /google/ gs://mygcpbucket/nextflow/f4/72c513e7a923fb0c80b30fc74c669d

1 reply

seems weird to see that generic permissions error

pmtempy

@pmtempy

I was running a Nextflow job with about 2k tasks on AWS Batch. Unfortunately, the Docker container for one of the processes contained an error (Exception in thread "Thread-1" java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper), and I had to kill the nextflow run. I guess I must have hit CTRL+C twice, because while the interactive nextflow CLI progress stopped, I'm still left with thousands of RUNNABLE jobs in AWS Batch.

Is there any quick way to remove them without potentially affecting other nextflow runs using the same compute queue?
How can I avoid similar issues in the future? I.e. how should I properly cancel a running nextflow run and make it clean up its jobs in Batch?

1 reply

Yasset Perez-Riverol

@ypriverol

hi all, i have a code like this:

ch_spectra_summary.map { tuple_summary ->
                         def key = tuple_summary[0]
                         def summary_file = tuple_summary[1]
                         def list_spectra = tuple_summary[1].splitCsv(skip: 1, sep: '\t')
                         .flatten{it -> it}
                         .collect()
                         return tuple(key.toString(), list_spectra) 
                        }
                   .groupTuple()
                   .set { ch_spectra_tuple_results}

is resturning something like

[supp_info.mzid.gz, [[supp_info.mzid, 2014-06-24, Velos005137.mgf, MGF, Velos005137.mgf, ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf]]]

but I would like to selec only the last column of the Csv

ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf

the result should be:

[supp_info.mzid.gz, [ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf]]

Sofia Stamouli

@sofstam

Hello,

I have the following python script in a nextflow process.

process update_image { 

script:
"""
#!/usr/bin/env python3 
import os, subprocess

subprocess.check_call(['singularity', 'pull', 'docker://busybox'])

}

The singularity is installed and is in the $PATH. The config file looks like:

singularity {
    singularity.enabled = true 
    singularity.autoMounts = true
}

However, I get the error: No such file or directory: 'singularity'. Any ideas what might be wrong here?

Where communities thrive

People

Repo info

Activity

Downloading plugin nf-amazon@1.3.4

nf-core/deepvariant v1.0"

Config Profile : test,docker