These are chat archives for nextflow-io/nextflow

2nd
Aug 2018
Adam Nunn
@bio15anu
Aug 02 2018 09:04 UTC
@pditommaso I'm trying to implement a conditional publishDir path, akin to a snippet that you suggested when someone asked this question previously:
publishDir path: {params.saveReference ? 'results' : null}, mode: 'copy'
but this fails for me, with this exception "Cannot invoke method complete() on null object"
is there a different way to do this in newer versions of NF ?
Paolo Di Tommaso
@pditommaso
Aug 02 2018 09:06 UTC
yes
Adam Nunn
@bio15anu
Aug 02 2018 09:07 UTC
ah perfect thank you!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 09:07 UTC
:+1:
rwgn23
@rwgn23
Aug 02 2018 11:14 UTC
Hi, I have a question about the whole Nextflow structure. So, at the moment we are discussing about whether we are using nextflow or snakemake for our future projects. At the moment most of us use snakemake, which sometimes gives them a hard time. Nextflow would be the solution for nearly all the problems they have. But one issue they have is that there is no option to implement a Nextflow script into another Nextflow script. The possibility to run a Nextflow script inside another one and get the resulting channels from one script to the parent one. For example you wrote a script to align reads with star. You are using this alignment in many different analyzing scripts. So you write an star alignment script and implement each one into all other scripts. Something like a module system. Is this maybe already possible or if not, do you plan on adding this as an feature?
Alexander Peltzer
@apeltzer
Aug 02 2018 11:17 UTC
rwgn23
@rwgn23
Aug 02 2018 11:25 UTC
ah thank you!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 11:37 UTC
component modules can be implemented very easily with plan bash wrappers
you can reuse across different projects including them as a gut submodule
if instead you are referring to module multiple (parallel) processes aggregation is a complete different story
Timur Shtatland
@tshtatland
Aug 02 2018 14:27 UTC
@pditommaso Thank you, your suggestion to use nextflow.config works. And thank you so much for your work on Nextflow!!!
Karin Lagesen
@karinlag
Aug 02 2018 14:34 UTC
hi, and good afternoon!
I am trying to send the results from a process to two channels
set pair_id, file("${pair_id}*_concat.fq.gz") into {reads; pilon_reads}
but I\m getting an Invalid output channel reference error
am I missing something bleedingly obvious here?
Félix C. Morency
@fmorency
Aug 02 2018 14:58 UTC
You can just do set pair_id, "${pair_id}*_concat.fq.gz" into reads, pilon_reads
Paolo Di Tommaso
@pditommaso
Aug 02 2018 15:00 UTC
the complete syntax is
    set val(pair_id), file("${pair_id}*_concat.fq.gz") into (reads, pilon_reads)
@tshtatland thanks
Mike Smoot
@mes5k
Aug 02 2018 20:31 UTC

Hi @pditommaso I'm finally back to debugging my AWS batch problem, which is that no jobs for my pipeline get submitted for code built from master whereas the same pipeline submits fine with version 0.30.2. I'm to the point of diffing nextflow logs. The only difference (apart from not submitting any jobs) is that master has a throttling executor and a different polling monitor:

main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
  [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
  [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
  [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: ParallelPollingMonitor

I've tried reproducing this problem with a simple pipeline to create a test case and I've had no luck. So master at least works for some pipelines. I'm just not sure where to look next to try and determine what's blocking my pipeline. Any ideas?

My code gets to this point

  [main] DEBUG nextflow.executor.Executor - Initializing executor: awsbatch
  [main] DEBUG nextflow.Session - >>> barrier register (process: test_mount)
  [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > test_mount -- maxForks: 2
  [main] DEBUG nextflow.script.ScriptRunner - > Await termination
  [main] DEBUG nextflow.Session - Session await

and hangs. In 0.30.2 it immediately follows the Session await with log messages from the [Task submitter]

Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:00 UTC
I think this an issue on latest version of aws batch executor, still not able to replicate
Mike Smoot
@mes5k
Aug 02 2018 21:01 UTC
Yeah, it's a tricky one. I'll keep banging away to see if I can find a reproducible test case.
Alexander Peltzer
@apeltzer
Aug 02 2018 21:01 UTC
@mes5k Is the one hanging a pipelin esubmitting many small jobs?
Just an idea
Because my pipeline that had issues with 0.31.0 did hang, while my student tried Paolos small rnaseq pipeline that ran through with 0.31.0
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:02 UTC
whats's your pipeline ?
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:04 UTC
even more weird, I've managed to run it with 0.31.0
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05 UTC
Ok
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:05 UTC
I think is a nondeterministic deadlock
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05 UTC
Looks like it, because its so tricky to reproduce
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:05 UTC
my favourite ones ... ;)
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05 UTC
;-)
I still have the EC2 instance that I used the last time - could maybe give you access to that tomorrow for having a look ...? !
I never got it to run on that one ... only when setting NXF_VER=0.30.2
n8, I'll have a lookt tomorrow
Mike Smoot
@mes5k
Aug 02 2018 21:08 UTC

Ok, I've risked life and limb to put this test code up:

${NF} run https://github.com/mes5k/nf-test.git -r master -latest -w s3://super-secret-bucket/nf-test-work

This consistently fails for me with master and works with 0.30.2.

Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:09 UTC
let me try
Mike Smoot
@mes5k
Aug 02 2018 21:10 UTC
@apeltzer these pipelines create a few small jobs - 5 to start.
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:11 UTC
now way
nextflow run https://github.com/mes5k/nf-test.git -r master -latest
N E X T F L O W  ~  version 0.31.0
Pulling mes5k/nf-test ...
 downloaded from https://github.com/mes5k/nf-test.git
Launching `mes5k/nf-test` [loving_colden] - revision: 8a738f4b5a [master]
1
2
3
4
5
[warm up] executor > awsbatch
[ff/985307] Submitted process > create_files (3)
[3a/296bc1] Submitted process > create_files (2)
[3d/0b5f53] Submitted process > create_files (1)
[12/8c45a9] Submitted process > create_files (4)
[fa/801711] Submitted process > create_files (5)
works
Mike Smoot
@mes5k
Aug 02 2018 21:12 UTC
hmmm
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:12 UTC
@mes5k what's your computer? linux?
Mike Smoot
@mes5k
Aug 02 2018 21:12 UTC
Yes. An AWS node running Ubuntu
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:12 UTC
how many cpus ?
Mike Smoot
@mes5k
Aug 02 2018 21:13 UTC
Just one
an m3.medium in AWS
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:14 UTC
I suspect it only show with one cpu
I'll make some tests tomorrow
Mike Smoot
@mes5k
Aug 02 2018 21:14 UTC
Sure, that sounds fine.
In the meantime I'll spin up a larger machine and keep experimenting!
Thanks!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:15 UTC
you are welcome
Alexander Peltzer
@apeltzer
Aug 02 2018 21:23 UTC
Also one t2.micro instance here
Mike Smoot
@mes5k
Aug 02 2018 21:32 UTC
Yup, just tried on an m3.large (2 cpus) and it worked!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:36 UTC
ah ah ..
if( cpus == 1 ) then println "error"
:joy:
Mike Smoot
@mes5k
Aug 02 2018 21:40 UTC
well, if it did actually print an error I'd be OK with it! :)