These are chat archives for nextflow-io/nextflow

20th
Jun 2018
sendivogius
@sendivogius
Jun 20 2018 08:54

Hi!

I have a pipeline which successfully finishes all processess but for unknown reason pipeline (as a whole) doesn't end.
I got following messages in log:

cze-20 10:42:58.165 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- The following nodes are still active:
[process] trimming
status=ACTIVE
port 0: (queue) OPEN; channel: -
port 1: (cntrl) OPEN; channel: $

[process] mapping
status=ACTIVE
port 0: (queue) OPEN; channel: -
port 1: (cntrl) OPEN; channel: $

Similar output is for all processess. What's the reason of that situation?

Paolo Di Tommaso
@pditommaso
Jun 20 2018 09:07
something is not properly terminating in your dataflow network
Paolo Di Tommaso
@pditommaso
Jun 20 2018 09:26
is the code available somewhere ?
sendivogius
@sendivogius
Jun 20 2018 12:14
No it isn't. Is there a way to debug it?
Paolo Di Tommaso
@pditommaso
Jun 20 2018 12:15
the problem is in the first process reported as status=ACTIVE or more likely a channel used as input
bisect the pipeline until you isolate the problem
sendivogius
@sendivogius
Jun 20 2018 13:13

I have found a problem. I changed

input = Channel.from(/*foo*/)

to

input = Channel.create()
input << /*foo*/

and forgot to invoke close on it.
Thanks!

Jemma Nelson
@fwip
Jun 20 2018 18:12
Is there any way to get a channel to repeat forever? My specific problem was with multiple input files and a single reference file that needed to be made available, and I solved it by combine()ing the two channels into one. But I could also see something like:
input:
file item from items
file reference from Channel.fromPath(params.reference).forever()
Félix C. Morency
@fmorency
Jun 20 2018 18:15
you can use .first()
input:
file items from items
file reference from Channel.fromPath(params.reference).first()
Jemma Nelson
@fwip
Jun 20 2018 18:17
Huh, that will repeat the first item forever? I would have expected that it would only yield 1 reference, and that the process would thus only run once (instead of as many times as there are inputs)
Félix C. Morency
@fmorency
Jun 20 2018 18:19
Me too but that's what Paolo told me to use :D
The naming is a bit confusing I agree
Jemma Nelson
@fwip
Jun 20 2018 18:20
Huh, wow, it looked like it worked. I wonder why that does that.
Jemma Nelson
@fwip
Jun 20 2018 18:33
Ah, searching in chat, it sounds like .first() returns the first item in the channel without consuming it
Paolo Di Tommaso
@pditommaso
Jun 20 2018 18:35
I need to add a pattern to example this ..
Mike Smoot
@mes5k
Jun 20 2018 18:38
@fwip @fmorency you've stumbled upon the difference between dataflow queues and dataflow values. dataflow queues are what your normal channels are and emit multiple values where each value can only be used once. dataflow values, on the other hand, are also channels, but can only hold a single value and very different from normal channels can be consumed multiple times. The first() operator only takes one value from a channel and because it's just one, that value gets magically transformed into a dataflow value, which you can use over and over. first() is a way to be explicit about this, but there are a few other cases when channels only produce one value that the dataflow valueness of a channel is magically inferred. Confusing?
Yes, indeed! Also very powerful. I don't think this is documented anywhere...
Paolo Di Tommaso
@pditommaso
Jun 20 2018 18:38
I don't think this is documented anywhere
the original sin :joy:
Mike Smoot
@mes5k
Jun 20 2018 18:39
Well, now it's documented in gitter
Paolo Di Tommaso
@pditommaso
Jun 20 2018 18:39
:+1:
Félix C. Morency
@fmorency
Jun 20 2018 18:39
@mes5k thanks. very cool.
Jemma Nelson
@fwip
Jun 20 2018 18:41
When reading the docs on Input; I didn't realize you could specify an input that was from a value, instead of from a channel.
it looks like I can just do file reference from file(params.reference), which is super readable
Paolo Di Tommaso
@pditommaso
Jun 20 2018 18:42
I never liked, file x from file y syntax :/
at some point it will change to path x from file y
Jemma Nelson
@fwip
Jun 20 2018 18:43
that sounds nice :)
Paolo Di Tommaso
@pditommaso
Jun 20 2018 18:45
good!