Nextflow community chat moved to Slack! https://www.nextflow.io/blog/2022/nextflow-is-moving-to-slack.html
pditommaso on master
Remove unnecessary change dir e… (compare)
jorgeaguileraseqera on 2845-add-support-for-labeltags
feature: add support for label… (compare)
jorgeaguileraseqera on 2845-add-support-for-labeltags
feature: add support for label/… feature: add support for label/… feature: add support for label/… and 3 more (compare)
pditommaso on 2881-if-nxf_offline-is-a-non-empty-string-then-offline-mode-is-enabled
process generate_readset {
tag "$sample_id"
cpus 48
input:
tuple val(read_name), val(chromosome1), val(chromosome2), val(cuteSV_pos1), val(cuteSV_pos2),
val(sniffle_pos1), val(sniffle_pos2),
path(cuteSV_vcf), path(sniffles_vcf) from vcf_input
output:
path 'complete_read_set.txt' into receiver
script:
"""
${bcftools_1_11} view --threads ${task.cpus} $cuteSV_vcf -r chr$chromosome1:$cuteSV_pos1-$cuteSV_pos2 > complete.txt
"""}
Remote resource not found: https://api.github.com/repos/PATH/TO/contents/main.nf
. What am I doing wrong?
Hey All,
I have a error relate to nextflow azurebatch. The first process using a default D4_v3 vm work alright, but the second process I fail to request a larger vm (I set it in queue, but apparently, it is not working, do I make some naive mistake?)
'''
Error executing process > 'secondprocess'
Caused by:
Cannot find a VM for task 'secondprocess' matching this requirements: type=Standard_D4_v3, cpus=16, mem=14 GB, location=eastus
'''
The config file I used:
process {
executor = 'azurebatch'
}
docker {
enabled = true
}
azure {
batch {
location = 'eastus'
accountName = 'xxxbatch'
accountKey = 'xxx'
autoPoolMode = true
allowPoolCreation = true
deletePoolsOnCompletion = true
deleteJobsOnCompletion = true
pools {
small {
autoScale = true
vmType = 'Standard_D4_v3'
vmCount = 5
maxVmCount = 50
}
large {
autoScale = true
vmType = 'Standard_D16_v3'
vmCount = 5
maxVmCount = 50
}
}
}
storage {
accountName = "xxx"
accountKey = "xxx"
}
}
process {
withName: firstprocess {
queue = 'small'
}
withName: secondprocess {
queue = 'large'
}
}
nextflow run
, despite specifying an unquota'd path with -w
aka -work-dir
. Any ideas? In this thread, Paolo suggests -w
is the solution… https://groups.google.com/g/nextflow/c/401Tp_6H57k/m/va8ACNeTAQAJ
$ nextflow run nf-core/viralrecon -w /users/xxx/test --help
N E X T F L O W ~ version 21.04.0
Pulling nf-core/viralrecon ...
Disk quota exceeded
How can I use this as the 'size' part of a groupTuple? I tried:
aligned_bams
.groupTuple(by:0,size:lane_calc)
But it did not like it - complained about the value type etc. All thoughts gladly received!
.collect()
the files from each tuple at the same time. Anyone has done this before? I've already read about each
and combine
, so my input channel is of the right format. The problem I have is .collect()
- not sure how to incorporate it in the input tuple.
Hi, in DSL2, do functions allow the passing in of channels? I was hoping to clean up a workflow where a similar sequence of operations is applied to channels, but it looks like channels passed into functions are demoted to plain Java collections.
The error reads:
No signature of method: java.util.LinkedHashMap.collectFile() is applicable for argument types: (LinkedHashMap, Script_77e02758$_collect_file_tuples_closure1) values: [[storeDir:null, sort:hash], Script_77e02758$_collect_file_tuples_closure1@54e2fe]
process {
withName: structural_alignment {
if (task.exitStatus in 140..143 )
"""
errorStrategy = 'retry'
cpus = {2 * task.attempt}
maxRetries = 5
"""
else
errorStrategy = 'retry'
maxRetries = 10
}
}
process {
withName: structural_alignment {
if (task.exitStatus in 140..143 ) {
errorStrategy = 'retry'
cpus = {2 * task.attempt}
maxRetries = 5
}
else {
errorStrategy = 'retry'
maxRetries = 10
}
}
Hi guys.
I have a simple doubt (I think). I just need to save my output process in an external directory so I'm using the publishDir
function for it. However I need 2 things: i) use variables (collected from input tuple); ii) Create the directory because it doesn't exist.
I successfully run it like this:
publishDir "${params.out_dir}/${tcga_project}/${tcga_barcode}/RSEM_quantification", mode: 'move'
input:
file STAR_bam_file from STAR_alignment_bam
set val(sample_UUID), val(tcga_barcode), val(tcga_project) from samples_ch_3
But It doesn't create the directory. So I tried switching to this:
outDir = file("${params.out_dir}/${tcga_project}/${tcga_barcode}/RSEM_quantification")
outDir.mkdirs()
publishDir outDir, mode: 'move'
input:
file STAR_bam_file from STAR_alignment_bam
set val(sample_UUID), val(tcga_barcode), val(tcga_project) from samples_ch_3
However it states the error of:No such variable: tcga_project
Any help in this situtation? Thanks :)
-profile docker
at the command line level, and instead I would like to have in my nextflow.config file profile = docker
such that it successfully imports all of the config options specified by the docker profile. thanks
collectFile
in my workflow to combine the results of several upstream processes, and setting storeDir
. However, when I resume my workflow (using AWS Batch executor), even though the upstream processes are correctly cached, the downstream process that consumes the output of collectFile
always re-executes. I've tried sorting the input to collectFile
with no luck.
How come
Channel
.fromFilePairs( "${params.reads}/*{1,2}*.fastq.gz", size: 2 )
.into { paired_reads }
does not work when the file starts with a number (like 7126-MS-1_100_S1_L005_R1_001.fastq.gz
and 7126-MS-1_100_S1_L005_R2_001.fastq.gz
or 21VR067049-211104_S1_L001_R1_001.fastq.gz
and21VR067049-211104_S1_L001_R2_001.fastq.gz
.collate()
in my pipeline and then an implementation of this which works elegantly. My problem is with the -resume
option. Some processes get cached, but the rest are re-scheduled. I am assuming the order changes from run to run before the collate operator, and thus assumes input has changed. Could this be it?
Channel.from(2,1,3,4).toSortedList().fromList()
Hello Group members, I am trying a sample nextflow script using Azure Blob storage for work directory. I am getting an error and couldn't understand if my setup has anything to do, appreciate your response. Here is the error when trying to run my sample script connecting to Azure Blob storage
nextflow run tutorial.nf -w az://nextflow-work/work
How to deal with the empty path channel in process (dsl2)? I saw a solution for empty channel of value (https://github.com/nextflow-io/patterns/blob/master/docs/process-when-empty.adoc)
nextflow.enable.dsl=2
params.a = '*.txt'
params.b = ''
params.c = ''
process test {
tag "test "
echo true
input:
path file1
path file2
val value1
"""
echo $file1
echo $file2
echo $value1
"""
}
workflow {
a_ch = Channel.fromPath(params.a)
b_ch = Channel.fromPath(params.b)
test(a_ch,b_ch,params.c)
}
The Error message:
N E X T F L O W ~ version 21.04.3
Launching `code/test2.nf` [modest_colden] - revision: b4078eef6a
Missing `fromPath` parameter
a_ch = params.a ? Channel.fromPath(params.a) : []
maybe?
Hi,
I'm trying to use Docker in Nextflow script and I'm getting the following error:
"connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Fr
un%2Fdocker.sock/v1.24/containers/create?name=nxf-8IoBF1Fj7HBmfAtOGRGpW5Fp": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'."
Does anyone have any advice on how to handle this?
:point_up: Edit: Hi,
I'm trying to use Docker in Nextflow script and I'm getting the following error:
"connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Fr
un%2Fdocker.sock/v1.24/containers/create?name=nxf-8IoBF1Fj7HBmfAtOGRGpW5Fp": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'."
Does anyone have any advice on how to handle this?
Hello,
I am trying to run nextflow inside a docker and from there: run other dockers in the host machine. For that I have built a docker image with nextflow and docker inside, as so:
FROM nextflow/nextflow
# Adds ability to run docker in the host
RUN amazon-linux-extras install docker -y
WORKDIR /nextflow
But when running the below line, it says Command error: /bin/bash: .command.sh: No such file or directory
docker run -v /var/run/docker.sock:/var/run/docker.sock custom/nextflow bash -c 'nextflow run hello -with-docker nextflow/examples'
Does anyone have an idea on how to fix this?
publish
command? I understand it is asynchronous by default, but I have a pipeline that runs for 7-10 days and some of the useful results are complete after the first day. Would be good if I could request the results from day 1 are "published" soon after completion.
init_Config (3)
terminated for an unknown reason -- Likely it has been terminated by the