These are chat archives for nextflow-io/nextflow

2nd
Aug 2018
Adam Nunn
@bio15anu
Aug 02 2018 09:04
@pditommaso I'm trying to implement a conditional publishDir path, akin to a snippet that you suggested when someone asked this question previously:
publishDir path: {params.saveReference ? 'results' : null}, mode: 'copy'
but this fails for me, with this exception "Cannot invoke method complete() on null object"
is there a different way to do this in newer versions of NF ?
Paolo Di Tommaso
@pditommaso
Aug 02 2018 09:06
yes
Adam Nunn
@bio15anu
Aug 02 2018 09:07
ah perfect thank you!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 09:07
:+1:
rwgn23
@rwgn23
Aug 02 2018 11:14
Hi, I have a question about the whole Nextflow structure. So, at the moment we are discussing about whether we are using nextflow or snakemake for our future projects. At the moment most of us use snakemake, which sometimes gives them a hard time. Nextflow would be the solution for nearly all the problems they have. But one issue they have is that there is no option to implement a Nextflow script into another Nextflow script. The possibility to run a Nextflow script inside another one and get the resulting channels from one script to the parent one. For example you wrote a script to align reads with star. You are using this alignment in many different analyzing scripts. So you write an star alignment script and implement each one into all other scripts. Something like a module system. Is this maybe already possible or if not, do you plan on adding this as an feature?
Alexander Peltzer
@apeltzer
Aug 02 2018 11:17
rwgn23
@rwgn23
Aug 02 2018 11:25
ah thank you!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 11:37
component modules can be implemented very easily with plan bash wrappers
you can reuse across different projects including them as a gut submodule
if instead you are referring to module multiple (parallel) processes aggregation is a complete different story
Timur Shtatland
@tshtatland
Aug 02 2018 14:27
@pditommaso Thank you, your suggestion to use nextflow.config works. And thank you so much for your work on Nextflow!!!
Karin Lagesen
@karinlag
Aug 02 2018 14:34
hi, and good afternoon!
I am trying to send the results from a process to two channels
set pair_id, file("${pair_id}*_concat.fq.gz") into {reads; pilon_reads}
but I\m getting an Invalid output channel reference error
am I missing something bleedingly obvious here?
Félix C. Morency
@fmorency
Aug 02 2018 14:58
You can just do set pair_id, "${pair_id}*_concat.fq.gz" into reads, pilon_reads
Paolo Di Tommaso
@pditommaso
Aug 02 2018 15:00
the complete syntax is
    set val(pair_id), file("${pair_id}*_concat.fq.gz") into (reads, pilon_reads)
@tshtatland thanks
Mike Smoot
@mes5k
Aug 02 2018 20:31

Hi @pditommaso I'm finally back to debugging my AWS batch problem, which is that no jobs for my pipeline get submitted for code built from master whereas the same pipeline submits fine with version 0.30.2. I'm to the point of diffing nextflow logs. The only difference (apart from not submitting any jobs) is that master has a throttling executor and a different polling monitor:

main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
  [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:5, maxPoolSize:5, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
  [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
  [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: ParallelPollingMonitor

I've tried reproducing this problem with a simple pipeline to create a test case and I've had no luck. So master at least works for some pipelines. I'm just not sure where to look next to try and determine what's blocking my pipeline. Any ideas?

My code gets to this point

  [main] DEBUG nextflow.executor.Executor - Initializing executor: awsbatch
  [main] DEBUG nextflow.Session - >>> barrier register (process: test_mount)
  [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > test_mount -- maxForks: 2
  [main] DEBUG nextflow.script.ScriptRunner - > Await termination
  [main] DEBUG nextflow.Session - Session await

and hangs. In 0.30.2 it immediately follows the Session await with log messages from the [Task submitter]

Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:00
I think this an issue on latest version of aws batch executor, still not able to replicate
Mike Smoot
@mes5k
Aug 02 2018 21:01
Yeah, it's a tricky one. I'll keep banging away to see if I can find a reproducible test case.
Alexander Peltzer
@apeltzer
Aug 02 2018 21:01
@mes5k Is the one hanging a pipelin esubmitting many small jobs?
Just an idea
Because my pipeline that had issues with 0.31.0 did hang, while my student tried Paolos small rnaseq pipeline that ran through with 0.31.0
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:02
whats's your pipeline ?
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:04
even more weird, I've managed to run it with 0.31.0
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05
Ok
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:05
I think is a nondeterministic deadlock
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05
Looks like it, because its so tricky to reproduce
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:05
my favourite ones ... ;)
Alexander Peltzer
@apeltzer
Aug 02 2018 21:05
;-)
I still have the EC2 instance that I used the last time - could maybe give you access to that tomorrow for having a look ...? !
I never got it to run on that one ... only when setting NXF_VER=0.30.2
n8, I'll have a lookt tomorrow
Mike Smoot
@mes5k
Aug 02 2018 21:08

Ok, I've risked life and limb to put this test code up:

${NF} run https://github.com/mes5k/nf-test.git -r master -latest -w s3://super-secret-bucket/nf-test-work

This consistently fails for me with master and works with 0.30.2.

Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:09
let me try
Mike Smoot
@mes5k
Aug 02 2018 21:10
@apeltzer these pipelines create a few small jobs - 5 to start.
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:11
now way
nextflow run https://github.com/mes5k/nf-test.git -r master -latest
N E X T F L O W  ~  version 0.31.0
Pulling mes5k/nf-test ...
 downloaded from https://github.com/mes5k/nf-test.git
Launching `mes5k/nf-test` [loving_colden] - revision: 8a738f4b5a [master]
1
2
3
4
5
[warm up] executor > awsbatch
[ff/985307] Submitted process > create_files (3)
[3a/296bc1] Submitted process > create_files (2)
[3d/0b5f53] Submitted process > create_files (1)
[12/8c45a9] Submitted process > create_files (4)
[fa/801711] Submitted process > create_files (5)
works
Mike Smoot
@mes5k
Aug 02 2018 21:12
hmmm
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:12
@mes5k what's your computer? linux?
Mike Smoot
@mes5k
Aug 02 2018 21:12
Yes. An AWS node running Ubuntu
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:12
how many cpus ?
Mike Smoot
@mes5k
Aug 02 2018 21:13
Just one
an m3.medium in AWS
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:14
I suspect it only show with one cpu
I'll make some tests tomorrow
Mike Smoot
@mes5k
Aug 02 2018 21:14
Sure, that sounds fine.
In the meantime I'll spin up a larger machine and keep experimenting!
Thanks!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:15
you are welcome
Alexander Peltzer
@apeltzer
Aug 02 2018 21:23
Also one t2.micro instance here
Mike Smoot
@mes5k
Aug 02 2018 21:32
Yup, just tried on an m3.large (2 cpus) and it worked!
Paolo Di Tommaso
@pditommaso
Aug 02 2018 21:36
ah ah ..
if( cpus == 1 ) then println "error"
:joy:
Mike Smoot
@mes5k
Aug 02 2018 21:40
well, if it did actually print an error I'd be OK with it! :)