I have installed click library in my machine
is conda like a docker/vm? does it maintain a separate environment? if so do I need to install "click" library in the bioconda library?
Hi, I asked a NextFlow question over here which has got a few votes so I gather other people are interested also: https://bioinformatics.stackexchange.com/questions/9099/passing-around-complex-metadata-in-nextflow. Answers much appreciated.
Hi, I am a nextflow newbie and was wondering if the following is possible: in my workflow I would like to keep an object defined in groovy (imagine a table) which keeps growing as steps of the pipeline finish (imagine each step adds a new column). I tested this concept but it is not working because nextflow executes things in parallel, not waiting untill the previous step finishes. This is what I'd like to do:
data = create_data()
Channel
.fromList( data.list_columns_relevant_for_process1() )
.set{ data_for_process1 }
run_process1( data_for_process1 )
// here I'd like the workflow to wait until process1 finishes
data.add_data( process1.out.data )
Channel
.fromList( data.list_columns_relevant_for_process2() )
.set{ data_for_process2 }
run_process2( data_for_process2 )
Hi there, does anyone know of any good resources about handling outputs in nextflow for R flexdashboards (https://rmarkdown.rstudio.com/flexdashboard/) ? Given that successfully running a flexdashboard rmarkdown file automatically opens a browser to mess about interactively with data, does anyone have any ideas how to approach this? I'm still quite new to nextflow and flexdashboard, so I can't see any obvious solutions to something like this, or any similar things to investigate further.
I'd be interested too. The way I've done it is to precompute something using nextflow & then filling in a separate process a RMarkdown file and render this as HTML and produce this as output. That makes sometimes sense, as the precompute steps are somewhat timeconsuming but we do explore the final output in a more interactive fashion / allow filtering in the dashboard HTML etc ...
But no nice resources about these types of things as far as I can tell
@apeltzer ah, that's a shame. We do a lot of precompute analysis in a previous step in our pipeline too, but a student working with us has created a really nice flexdashboard which runs on the outputs of these previous steps and lets us select samples (genomics data pipeline) to view plots for those samples. We haven't got the flexdashboard part running seamlessly from commandline yet, so I'm still not 100% on the output format of flexdashboards myself, but I'm attempting to pull together a process in our .nf to integrate it as it's coming along. Is there any chance your project is open and I could have a peek at how you're handling it? It sounds like you're doing something similar...
But yes, it sounds very similar to what we are doing and there are some other project where I can check if this is already open
Similar approach could be to generate a standardized output + provide some app in RShiny to read that file and make interaction possible
(other project we do this, writing a RData object and then parsing this in the Shiny App for visual / interactive exploration)
Hello all, I was hoping to get some help to figure out why the
bin dir isn't getting copied over correctly when I'm using pytest-workflow to test some workflow(It makes sym-links under the hood) nf-core/rnaseq#546
Hi all,
Wondering if anyone has observed this behaviour when pulling a container via config process withLabel.
Config:
process {
..
..
withLabel: multiqc {
container = 'docker://barryd237/week1:test'
}
}
singularity.enabled = true
singularity.autoMounts = trueThe container is successfully pulled, however the naming of the file throws a java.nio.file.NoSuchFileException: /data/../work/singularity/barryd237-week1-test.img.pulling.1611136739603 error. The container is downloaded as barryd237-week1-test.pulling.1611136739603.img.
I can see the discrepancy in how the container was named and how nextflow looks for it, but I don't know what the root cause is.
Hi all, I have two processes. first one - findFileRelationships - which produces three text files
and second one - launchSubmissionValidation which should take each file and run a command
I get this error when I pass the input variable
assay_file
Error: Got unexpected extra arguments (7_assay_relation.txt 8_assay_relation.txt)
it seems my
assay_file variable contains all three file names.
How can I make it a channel and execute them in parallel ?
* Find the relationship between RESULT file(s) and PEAK file(s) from Submission.px file
*/
process findFileRelationships {
output:
file '*_assay_relation.txt' into assay_files
script:
"""
python3 $workflow.launchDir/scripts/readPX.py rel -i $params.submission_px
"""
}
/*
* Launch the validation for all the results
*/
process launchSubmissionValidation {
conda 'submission-tool-validator=1.0.1 click'
errorStrategy 'retry'
maxErrors 3
input:
file assay_file from assay_files
output:
file 'stv_output.txt' into validation_results
script:
"""
python3 $workflow.launchDir/scripts/readPX.py validation -f $assay_file -d $params.data_base_dir
"""
}assay_files.view { "value: $it" } gives me value: [/Users/hewapathirana/Desktop/validation/work/5f/866e63d592a56dd5ecbe92c9e7622e/6_assay_relation.txt, /Users/hewapathirana/Desktop/validation/work/5f/866e63d592a56dd5ecbe92c9e7622e/7_assay_relation.txt, /Users/hewapathirana/Desktop/validation/work/5f/866e63d592a56dd5ecbe92c9e7622e/8_assay_relation.txt]
I have 4 modules with split of related processes. but some steps in the 4th file can start as soon as some basic ones in the 1st finish, while others have to first pass through 2 & 3.
So I was wondering, is there a way to import all the functions from a module? So how I can directly access individual process output from a module without first having to wait for the entire module to finish?
Is there an import * from './module' like syntax? Because the alternative would be something like:
include { func1, func2, func3, funcN, func20+ } from './moduleName'
So I was wondering, is there a way to import all the functions from a module? So how I can directly access individual process output from a module without first having to wait for the entire module to finish?
Is there an import * from './module' like syntax? Because the alternative would be something like:
include { func1, func2, func3, funcN, func20+ } from './moduleName'
Another issue I am having is that some steps always cache succesfully, but many always fail to cache.
Whilst the only thing they do is download a file based on the input, this file is garanteed to never change for the same input. I download it as an archive, and output the decompressed file
Whilst the only thing they do is download a file based on the input, this file is garanteed to never change for the same input. I download it as an archive, and output the decompressed file
I think anonymous imports have been deprecated @YPHa
Check the DSL2 migration notes here: https://www.nextflow.io/docs/latest/dsl2.html
For my complicated modules I just have one include per line
Try cache = 'deep' to cache based on file input instead of timestamps (this is slower)
5 replies
Check cache directives for a full list and explanation: https://www.nextflow.io/docs/latest/process.html
Nextflow version:
v20.10.0, build 5430
I've got a fairly complicated workflow (somatic and germline variant calling, transcription quantification, MHC binding predictions, etc.) that seems to work well on my local SLURM cluster. I'm trying to run the same workflow on AWS Batch and am running into an error regarding a process,
get_fastqs, having already been used and to use an alias (e.g. include { foo as bar } from ...) if using it again. This doesn't make any sense to me because get_fastqs is being called in a differing context each time. Are there any known limitation on AWS Batch regarding reusing processes that don't exists on other executors? I'll reply to this comment with relevant information.
6 replies
and any idea how to debug/fix?
Hi, I recently started getting an error when running nextflow pipelines that use singularity. The error occurs when
singularity exec is called with `--vm-cpu flag. The error occurs outside of nextflow, so it may be a singularity issue. Has nextflow always called singularity using --vm-cpu flag? I'm on centos 7. thanks