@pditommaso I'm trying to implement a conditional publishDir path, akin to a snippet that you suggested when someone asked this question previously:
publishDir path: {params.saveReference ? 'results' : null}, mode: 'copy'
but this fails for me, with this exception "Cannot invoke method complete() on null object"
is there a different way to do this in newer versions of NF ?
yes
ah perfect thank you!
:+1:
Hi, I have a question about the whole Nextflow structure. So, at the moment we are discussing about whether we are using nextflow or snakemake for our future projects. At the moment most of us use snakemake, which sometimes gives them a hard time. Nextflow would be the solution for nearly all the problems they have. But one issue they have is that there is no option to implement a Nextflow script into another Nextflow script. The possibility to run a Nextflow script inside another one and get the resulting channels from one script to the parent one. For example you wrote a script to align reads with star. You are using this alignment in many different analyzing scripts. So you write an star alignment script and implement each one into all other scripts. Something like a module system. Is this maybe already possible or if not, do you plan on adding this as an feature?
ah thank you!
component modules can be implemented very easily with plan bash wrappers
you can reuse across different projects including them as a gut submodule
if instead you are referring to module multiple (parallel) processes aggregation is a complete different story
@pditommaso Thank you, your suggestion to use nextflow.config works. And thank you so much for your work on Nextflow!!!
hi, and good afternoon!
I am trying to send the results from a process to two channels
set pair_id, file("${pair_id}*_concat.fq.gz") into {reads; pilon_reads}
but I\m getting an Invalid output channel reference error
am I missing something bleedingly obvious here?
You can just do
set pair_id, "${pair_id}*_concat.fq.gz" into reads, pilon_reads
the complete syntax is
set val(pair_id), file("${pair_id}*_concat.fq.gz") into (reads, pilon_reads)
@tshtatland thanks
Hi @pditommaso I'm finally back to debugging my AWS batch problem, which is that no jobs for my pipeline get submitted for code built from master whereas the same pipeline submits fine with version 0.30.2. I'm to the point of diffing nextflow logs. The only difference (apart from not submitting any jobs) is that master has a throttling executor and a different polling monitor:
main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
[main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
[main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
[main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: ParallelPollingMonitorI've tried reproducing this problem with a simple pipeline to create a test case and I've had no luck. So master at least works for some pipelines. I'm just not sure where to look next to try and determine what's blocking my pipeline. Any ideas?
My code gets to this point
[main] DEBUG nextflow.executor.Executor - Initializing executor: awsbatch
[main] DEBUG nextflow.Session - >>> barrier register (process: test_mount)
[main] DEBUG nextflow.processor.TaskProcessor - Creating operator > test_mount -- maxForks: 2
[main] DEBUG nextflow.script.ScriptRunner - > Await termination
[main] DEBUG nextflow.Session - Session awaitand hangs. In 0.30.2 it immediately follows the Session await with log messages from the [Task submitter]
I think this an issue on latest version of aws batch executor, still not able to replicate
Yeah, it's a tricky one. I'll keep banging away to see if I can find a reproducible test case.
@mes5k Is the one hanging a pipelin esubmitting many small jobs?
Just an idea
Because my pipeline that had issues with 0.31.0 did hang, while my student tried Paolos small rnaseq pipeline that ran through with 0.31.0
whats's your pipeline ?
even more weird, I've managed to run it with 0.31.0
Ok
I think is a nondeterministic deadlock
Looks like it, because its so tricky to reproduce
my favourite ones ... ;)
;-)
I still have the EC2 instance that I used the last time - could maybe give you access to that tomorrow for having a look ...? !
I never got it to run on that one ... only when setting
NXF_VER=0.30.2
n8, I'll have a lookt tomorrow
Ok, I've risked life and limb to put this test code up:
${NF} run https://github.com/mes5k/nf-test.git -r master -latest -w s3://super-secret-bucket/nf-test-workThis consistently fails for me with master and works with 0.30.2.
let me try
@apeltzer these pipelines create a few small jobs - 5 to start.
now way
nextflow run https://github.com/mes5k/nf-test.git -r master -latest
N E X T F L O W ~ version 0.31.0
Pulling mes5k/nf-test ...
downloaded from https://github.com/mes5k/nf-test.git
Launching `mes5k/nf-test` [loving_colden] - revision: 8a738f4b5a [master]
1
2
3
4
5
[warm up] executor > awsbatch
[ff/985307] Submitted process > create_files (3)
[3a/296bc1] Submitted process > create_files (2)
[3d/0b5f53] Submitted process > create_files (1)
[12/8c45a9] Submitted process > create_files (4)
[fa/801711] Submitted process > create_files (5)
works
hmmm
@mes5k what's your computer? linux?
Yes. An AWS node running Ubuntu
how many cpus ?
Just one
an m3.medium in AWS
I suspect it only show with one cpu
I'll make some tests tomorrow
Sure, that sounds fine.
In the meantime I'll spin up a larger machine and keep experimenting!
Thanks!
you are welcome
Also one t2.micro instance here
Yup, just tried on an m3.large (2 cpus) and it worked!
ah ah ..
if( cpus == 1 ) then println "error"
:joy:
well, if it did actually print an error I'd be OK with it! :)