Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html
Command error:
Execution failed: generic::failed_precondition: while running "nf-2f69f94540149df9eda94c49022f51cc-main": unexpected exit status 1 was not ignored.command.out .command.err ..command.trace
nextflow-21.10.5 - any workflow does this including hello-world
gsutil cat gs://mygcpbucket/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/google/logs/output
/bin/bash: /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log: Permission denied
+ trap 'err=$?; exec 1>&2; gsutil -m -q cp -R /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log || true; [[ $err -gt 0 || $GOOGLE_LAST_EXIT_STATUS -gt 0 || $NXF_DEBUG -gt 0 ]] && { ls -lah /nextflow/f4/72c513e7a923fb0c80b30fc74c669d || true; gsutil -m -q cp -R /google/ gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d; } || rm -rf /nextflow/f4/72c513e7a923fb0c80b30fc74c669d; exit $err' EXIT
+ err=1
+ exec
+ gsutil -m -q cp -R /nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log gs://truwl-internal-inputs/nextflow/f4/72c513e7a923fb0c80b30fc74c669d/.command.log
+ [[ 1 -gt 0 ]]
+ ls -lah /nextflow/f4/72c513e7a923fb0c80b30fc74c669d
total 40K
drwxr-xr-x 3 root root 4.0K Dec 10 23:51 .
drwxr-xr-x 3 root root 4.0K Dec 10 23:51 ..
-rw-r--r-- 1 root root 3.3K Dec 10 23:51 .command.log
-rw-r--r-- 1 root root 5.3K Dec 10 23:51 .command.run
-rw-r--r-- 1 root root 36 Dec 10 23:51 .command.sh
drwx------ 2 root root 16K Dec 10 23:50 lost+found
+ gsutil -m -q cp -R /google/ gs://mygcpbucket/nextflow/f4/72c513e7a923fb0c80b30fc74c669d
1 reply
I was running a Nextflow job with about 2k tasks on AWS Batch. Unfortunately, the Docker container for one of the processes contained an error (Exception in thread "Thread-1" java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper), and I had to kill the nextflow run. I guess I must have hit CTRL+C twice, because while the interactive nextflow CLI progress stopped, I'm still left with thousands of RUNNABLE jobs in AWS Batch.
Is there any quick way to remove them without potentially affecting other nextflow runs using the same compute queue?
How can I avoid similar issues in the future? I.e. how should I properly cancel a running nextflow run and make it clean up its jobs in Batch?
1 reply
ch_spectra_summary.map { tuple_summary ->
def key = tuple_summary[0]
def summary_file = tuple_summary[1]
def list_spectra = tuple_summary[1].splitCsv(skip: 1, sep: '\t')
.flatten{it -> it}
.collect()
return tuple(key.toString(), list_spectra)
}
.groupTuple()
.set { ch_spectra_tuple_results}[supp_info.mzid.gz, [[supp_info.mzid, 2014-06-24, Velos005137.mgf, MGF, Velos005137.mgf, ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf]]]
ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf
[supp_info.mzid.gz, [ftp://ftp.ebi.ac.uk/pride-archive/2014/06/PXD001077/Velos005137.mgf]]
Hello,
I have the following python script in a nextflow process.
process update_image {
script:
"""
#!/usr/bin/env python3
import os, subprocess
subprocess.check_call(['singularity', 'pull', 'docker://busybox'])
}The singularity is installed and is in the $PATH. The config file looks like:
singularity {
singularity.enabled = true
singularity.autoMounts = true
}However, I get the error: No such file or directory: 'singularity'. Any ideas what might be wrong here?
Hello,
I have the following python script in a nextflow process.
process update_image { script: """ #!/usr/bin/env python3 import os, subprocess subprocess.check_call(['singularity', 'pull', 'docker://busybox']) }The singularity is installed and is in the $PATH. The config file looks like:
singularity { singularity.enabled = true singularity.autoMounts = true }However, I get the error:
No such file or directory: 'singularity'. Any ideas what might be wrong here?
Try the following:
singularity {
enabled = true
autoMounts = true
}
I have a regular expression in my parameters
params {
normal_name = /^NF-.*-3.*/
}that I use to match in my workflow elsewhere
def split_normal = branchCriteria {
normal: it.name =~ params.normal_name
}I'm trying to override this parameter as a CLI argument with --normal_name '/^NF-.*-4.*/' but then it's treated as a string in the workflow instead. Is there a good way to handle this, perhaps by compiling the parameter in the workflow?
1 reply
Hi, I'm trying to get the sarek pipeline work on our hpc cluster.
here is my command
nextflow run nf-core/sarek -r 2.7.1 -profile singularity -c nfcore_ui_hpc.config --input '/Users/rsompallae/projects/janz_lab_wes/fastqs_1.tsv' --genome mm10
and the error I get is
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 32 bytes for AllocateHeap
An error report file with more information is saved as:
I feel like this has to do with some parameter adjustment but I'm not sure how to fix it.
Thanks in advance for your help
I would like to use split -l 1 on an input file and then emit each small file out on its own with a tuple maintaining metadata for the initial file that was split, instead of having all of them contained in that one field of the tuple.
something like this:
process split {
input:
tuple path(run), val(plateid), path(file) from ex_ch
output:
tuple path(run), val(plateid), path("file_*") into parse_ch
script:
"""
tail -n +17 $file > sample_lines.csv
split -l 1 -d sample_lines.csv file_
"""
}But this should ideally emit the number of lines in sample_lines.csv as tasks. With this setup they're all caught into a single channel and you get a tuple that looks like:
['/path/to/run', 'plate_id', 'file_1, file_2, file_3, file_4']Anyone have a quick way to do this? Maybe it's just a .multimap / .map but I can't seem to get it right.
1 reply
.splitCsv operator. Now this works nicely when running this locally or on a cluster, but now I tried to move this to AWS S3. Is there a way to retreive the s3 directory location where the files are stored and put it in the CSV file? Or how would I approach this?
file "*.rds" into group_chunks_list. I tried then fromList but without luck. I am not sure if I am missing smth.
5 replies
Error: Could not find or load main class run i sometimes get this error immediately when running nextflow, and no files are produced by nextflow. what does this error actually mean? i can see that the .nf script i am calling is accessible and the nextflow executable is also accessible.
file() a path to gs://? It's not liking it at the moment, and I'm guessing it might be user error. Works fine for file() in a process
gs
singularity is there a way to get it to look for files in a specific local folder without specifying the full path in each process.container directive? I tried setting the singularity.cacheDir to one which contained my image files but unfortunately that didn't seem to work. I specified the image file name including the extension (image_name.img) perhaps that is the issue?
6 replies
first() warning (WARN: The operator first is useless when applied to a value channel which returns a single value by definition)?
I'm having a strange issue, and wonder if the way I'm using DSL2 modules is not correct. I have a workflow defined in one file with the following structure:
workflow foo {
take:
inputsPath
main:
a(inputsPath)
b(inputsPath)
c(inputsPath, a.out, b.out)
}a, b and c are just processes that run a python script over each file in the directory inputsPath and store the results in an output directory
Then in main.nf I have:
include { foo as foo1 } from './file.nf'
include { foo as foo2 } from './file.nf'
include { foo as foo3 } from './file.nf'
workflow {
foo1(Channel.fromList(['inputs1-1', 'inputs1-2']))
foo1(Channel.fromList(['inputs2-1', 'inputs2-2']))
foo1(Channel.fromList(['inputs3-1', 'inputs3-2']))
}The problem is that when the foo workflow executes the c process, the inputs are mixed up, i.e. it might be called with the output of a run on inputs1 and b run on inputs2.
This seems like it should be impossible, as the processes run by foo1 should only be looking at inputs/outputs associated with inputs1, correct?
foo it imports separate instances of the workflow, but not separate instances of the processes, and so a.out doesn't necessarily refer to the same a process that was started by the current workflow instance?
hi all
I am having a issue with staging multiple files and I have reported here - https://github.com/nextflow-io/nextflow/issues/1364#issuecomment-999285314
Wondering if anyone else in the gitter community has faced this issue and what was the workaround
I am using NF with AWSBatch and as suggested there, I have tried beforeScript: 'ulimit -s unlimited' but it does not seem to work
Thanks in advance
Hello all, I am facing an error trying to use nextflow with azure batch. Currently, I am using the nf-core/sarek pipeline and at the begining of the pipeline it gives me the error below:
Error executing process > 'MapReads (1543_18-1)'
Caused by:
Cannot find a VM for task 'MapReads (1543_18-1)' matching this requirements: type=Standard_E8_v3, cpus=16, mem=60 GB, location=brazilsouth
Any help will be appreciated,
Zilton.
4 replies
Nextflow expects that data paths are defined system wide, and your Singularity images need to be created having the mount paths defined in the container file system.
Hello people,
I could solve the previous error on Azure Batch and now another problem appear:
Caused by:
At least one value of specified task container settings is invalid
Command executed:
alleleCounter --version &> v_allelecount.txt 2>&1 || true
bcftools --version &> v_bcftools.txt 2>&1 || true
bwa version &> v_bwa.txt 2>&1 || true
cnvkit.py version &> v_cnvkit.txt 2>&1 || true
configManta.py --version &> v_manta.txt 2>&1 || true
configureStrelkaGermlineWorkflow.py --version &> v_strelka.txt 2>&1 || true
echo "2.7.1" &> v_pipeline.txt 2>&1 || true
echo "21.10.6" &> v_nextflow.txt 2>&1 || true
snpEff -version &> v_snpeff.txt 2>&1 || true
fastqc --version &> v_fastqc.txt 2>&1 || true
freebayes --version &> v_freebayes.txt 2>&1 || true
freec &> v_controlfreec.txt 2>&1 || true
gatk ApplyBQSR --help &> v_gatk.txt 2>&1 || true
msisensor &> v_msisensor.txt 2>&1 || true
multiqc --version &> v_multiqc.txt 2>&1 || true
qualimap --version &> v_qualimap.txt 2>&1 || true
R --version &> v_r.txt 2>&1 || true
R -e "library(ASCAT); help(package='ASCAT')" &> v_ascat.txt 2>&1 || true
samtools --version &> v_samtools.txt 2>&1 || true
tiddit &> v_tiddit.txt 2>&1 || true
trim_galore -v &> v_trim_galore.txt 2>&1 || true
vcftools --version &> v_vcftools.txt 2>&1 || true
vep --help &> v_vep.txt 2>&1 || true
scrape_software_versions.py &> software_versions_mqc.yaml
Command exit status:
-
Command output:
(empty)
Work dir:
az://genomas-raros/work/11/f701b3eb21426f510b935f9e75da58Any ideas?
Thanks in advance,
Zilton.