I am just running one set of paired reads to see how this works, and I can't figure out how to run the pipeline without downloading the reads first.
Do I need to download the reads to my computer first in order to get a 'path to fastq file', in order to create the csv sheet for implementing the pipeline?
The paired sample reads are here from ENA
https://www.ebi.ac.uk/ena/browser/view/SRS8234006
I can't seem to figure out if there is a 'path to file' directly into ENA without downloading the fastq files themselves...if this is even possible?
1 reply
Hi!
I'd like to have a profile in which I can specify some processes to be skipped:
process.withLabel: 'research' {
when = false
}But that doesn't work, since when is a declaration, not a directive.
I can see a workaround, such as declaring a params.research = true and then, in each process, write when: (!params.research) but the logic polutes a bit my processes.
Any suggestion? Thanks!
Hi All,
Has anyone used the VEP Docker image in Nextflow with additional reference files? The VEP Docker image requires the reference files to be mounted to the image from a local directory. I can do this easily outside of Nextflow but I am not sure how to tell Nextflow to mount a directory when launching the docker image.
I also want to eventually do this in an AWS Batch environment. Right now I'm just doing testing on a single EC2 instance. Thanks!
1 reply
1 reply
.vep folder
.vep folder)
I'm looking for recommendations on how to combine tuple channels.
I have a process that outputs multiple bam files and their associated sample names as a tuple channel, e.g.: tuple val(names), path(bams) into bams_ch. Viewing the bams channel gives output like:
[sample1, sample1_bwM_sort.bam]
[sample2, sample2_bwM_sort.bam]My next process compares those two bam files and outputs the comparison as a single output file. Essentially, the command is something like compareBams --input sample1_bwM_sort.bam sample2_bwM_sort.bam --output sample1_sample2_bwM_sort_compare.txt.
What I can't figure out is how to structure the input channel for this bam comparison process so that 1) it brings in both bam files at once so it can compare them; and 2) I can combine the two sample names, separated by '_', to use them as a prefix for the output file.
3 replies
git submodule. Would that work with Nextflow? I see @pditommaso mentioned something reminiscent of this in 2019, but I'm wondering if this would work.
Hello,
How can I add native code in a process? I tried exec process block but nothing is printed in the output. For example:
process test {
tag "$name"
input:
set val(name), file (json) from test_ch
exec:
Gson gson = new Gson();
System.out.println("test");
}I cannot figure out what might go wrong here. Any help?
How do you perform your processings, save the results in a folder, clean the crowded work directory every now and then, and still, resume the analytics and avoid processing the samples that were already processed?
AFAIK when the work directory is deleted, "-resume" cannot work and storeDir is an hazardous option.
4 replies
"Pipeline failed: Object of class 'groovyx.gpars.dataflow.DataflowVariable' does not support 'splitter' methods"
process get_sample_sizes {
input:
val samplePath
output:
stdout emit: sampleSizes
shell:
'''
du !{samplePath} | sort -rh | awk -F / '{print $NF","$1}' | sed -e 's/[[:space:]]*$//'
'''
}
workflow my_workflow {
main:
get_sample_sizes( samplePath)
Channel.from( get_sample_sizes.out.sampleSizes )
.splitCsv(header: ['sampleName', 'sampleSize'] )
.map{ row-> tuple(row.sampleName, row.sampleSize) }
.set { sample_size_ch }
}
setonly allows one channel to be set, if I saw that correctly.
Does anyone have a clean solution for adding directories to a channel based on the contents of that directory for flag files? I want to only use a channel for input into the pipeline if it does not contain flags for completion (for data retention purposes the directories that are complete must stay there).
Something like this:
Channel.fromPath("params.target_dir/*", type:'dir')
.filter(!it.contains("complete.flag")
.tap{jobs_to_run_ch}
If I run the pipeline with this command:
nextflow run main.nf -c test.confI would like to access all the parameters in
test.conf params{} for the creation of a README file, ideally without typing out each parameter.Thank you!
I'd like to filter a channel of bam files for those having at least a certain number of reads, to stop downstream processes from giving me an error (not enough reads to call copy number changes, or use ngscheckmate for sample matching).
What is the easiest way to do that in nextflow?
Hi All,
I'm trying to bind mount a directory when launching a docker image in Nextflow. Based on reading https://www.nextflow.io/docs/latest/config.html
this seems doable using the Scope docker "temp" settings. But I'm having trouble getting it to work. I only want to do this for a specific process.
In my process I'm trying
input:
path(ref_genome)
docker.temp = "${ref_genome}"But this returns an error saying ref_genome is not a variable. Ref_genome is simply a string pointing to a directory in an S3 bucket. Is this not allowed?
Thanks!
Hello,
I posted before regarding how to use a file from exec process block. I got a bit further but not much.
process test {
tag "$name"
input:
set val(name), file (json_file) from test_ch.view()
exec:
Gson gson = new Gson();
Reader reader = new FileReader(json_file);
Analysis result = gson.fromJson(reader, Analysis.class);But I get the error:
Caused by:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was BEGIN_ARRAY at line 1 column 2 path $
Source block:
Gson gson = new Gson();
Reader reader = new FileReader(json_file);
Analysresultat analysresultat = gson.fromJson(reader, Analysresultat.class);This code works fine in a java environment and I cannot figure out what I might do wrong here. I also tried reading the json file as a string but it did not work either. Any help?
I have a quick question regarding running my nextflow pipeline. I have some test data inside the nextflow directory. Now if I wanna run tests with this data and I specify a relative path, I get the following error
nextflow run main.nf -resume \
-with-report "../nextflow_reports/test_report.html" \
-with-timeline "../nextflow_reports/test_timeline.html" \
--grm_plink_input "./test_data/input/nfam_100_nindep_0_step1_includeMoreRareVariants_poly.{bed,bim,fam}" \
--phenoFile "./test_data/input/pheno*.txt" \
--phenoCol "y_binary" \
--covarColList "x1,x2" \
--bgen_filebase "genotype_100markers" \
--bgen_path "./test_data/input" \
--sampleFile "./test_data/input/samplefile_test_input.txt" \
--outdir "../saige_test_out"
Not a valid path value: ./test_data/input/samplefile_test_input.txtWhat's the correct way to define input files with relative file paths?
3 replies
I feel like I must be missing something basic about DSL2 syntax. I'm in the process of converting a workflow from DSL -> DSL2 and have run into an issue where processes only use the first item in a channel. I've figured out that it only happens when I have more than one input into the process and removing the additional inputs (e.g., a val) allows the process to use all items in the channel. I'm using v21.04.0 (installed through conda) and here's a super simple minimum working example that reproduces the issue on my machine:
test.nf
process TEST {
input:
tuple val(id), val(other)
val test // <- commenting this out and the corresponding line in main.nf executes the process twice
output:
path "test.txt", emit: test
script:
"""
touch test.txt
"""
}main.nf
#!/usr/bin/env nextflow
nextflow.enable.dsl = 2
include { TEST } from './test.nf'
ch_test = Channel.from( ["abc", "123"], ["def", "456"])
workflow {
TEST (
ch_test,
Channel.from( "iii" )
)
TEST.out.view()
}Running it
nextflow run main.nfThanks in advance for any help!
1 reply
