These are chat archives for nextflow-io/nextflow

16th
Nov 2015
Johann Visagie
@wjv
Nov 16 2015 13:35
Hi there. I have a … probably naïve question: In the “basic concepts” section of the documentation, Nextflow channels are described as asynchronous FIFO queues…
… so I created a simple test workflow with three processes, each one getting its STDIN from the previous one’s STDOUT
So in shell parlance, a | b | c
However, running that workflow locally in Nextflow did not result in a, b and c running concurrently; instead, b waited for a to complete (etc.)
I note that actual data transfer was via files written in the .work directory, not a named pipe. Is there a way to construct a workflow to execute such piped processes concurrently?
Paolo Di Tommaso
@pditommaso
Nov 16 2015 13:56
@wjv The process are executed in parallel but as long as there's a dependency between the tasks, c need to wait and output from b to be executed, and b from a
regarding the named pipe, take in account that nextflow is designed to run a workflow in a distributed manner in a cluster of computer, so it cannot use a named pipes to share data between different nodes
Johann Visagie
@wjv
Nov 16 2015 13:59
Hmm, OK, thanks. That’s a pity for my particular application.
Yep, I understand the reasoning there.
Paolo Di Tommaso
@pditommaso
Nov 16 2015 19:49
@wjv What's your use case ? if I can ask.