These are chat archives for nextflow-io/nextflow

20th
Jul 2016
Alexander Junge
@JungeAlexander
Jul 20 2016 13:05
reuse
Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:06
?
Alexander Junge
@JungeAlexander
Jul 20 2016 13:43

Hi there, I am struggling with the following use-case: A process receives a) multiple files from a channel A and b) a single file B and does something using each file from A together with B. The way I got this to run is:

process foo {
  input:
    file a from A
    each B

  """
  cat $a
  cat $B
  """
}

However, the following warning is printed:

WARN: Using queue channel on each parameter declaration should be avoided -- take in consideration to change declaration for each: 'B' parameter

What would be a better may to do this?

Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:45
ok, I'm waiting you complete the snipped above ..
Alexander Junge
@JungeAlexander
Jul 20 2016 13:46
Thanks :) confused shift+enter with enter
Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:48
ok, but you said that B is a single file, why you are using each on it ?
Alexander Junge
@JungeAlexander
Jul 20 2016 13:50
true but when using file B instead of each B, only the first element of A seems to be processed
Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:50
that's the point !
try this
process foo {
  input:
    file a from A
    file b from B.first()

  """
  cat $a
  cat $B
  """
}
the problem is that B is supposed to bring many files, thus by default it it will launch an execution for any pair (a,b) that are received
as soon a channel does not contain any data stops the execution
Alexander Junge
@JungeAlexander
Jul 20 2016 13:53
this works! and no warning! :)
yes, true. But I failed to see that first is usful in this context
Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:54
instead putting .first() you declares that you are expecting just one input for channel B, thus it will apply it as many times as the process is executed
this is quite tricky at the beginning
the general logic is that there should be a new input to be received from any channel to trigger a new run
except when they are singleton channels, i.e. that emit always the same value
Alexander Junge
@JungeAlexander
Jul 20 2016 13:57
true, that broke my neck. was searching the doc for terms like 'repeat' but failed to see that first actually does what I need
Paolo Di Tommaso
@pditommaso
Jul 20 2016 13:58
ok, just ask when you are in trouble :)
I think we put it in this FAQ
Alexander Junge
@JungeAlexander
Jul 20 2016 14:02
jep, will do! thanks for the quick help.
True it's item 8. but I haven't seen the FAQ, unfortunately. Perhaps you could link to it from website and/or documentation?
Paolo Di Tommaso
@pditommaso
Jul 20 2016 14:54
I will, thanks
Raymond Lim
@raylim
Jul 20 2016 22:48
re. including of .nf files, is this a planned feature? I don't see how it's possible to write large workflows without such a feature
also, is there a way to reuse a process multiple times in a workflow?
Paolo Di Tommaso
@pditommaso
Jul 20 2016 22:55
Yes, we plan to add this feature to nextflow though not in the immediate future.
In our experience however is not such a problem writing big pipeline without it, look for example at this
Raymond Lim
@raylim
Jul 20 2016 23:00
that doesn't look very maintainable...
Paolo Di Tommaso
@pditommaso
Jul 20 2016 23:01
The idea in nextflow is that the unit of reusability is the process that allows you to include/execute any command/tool, thus complex commands can be decoupled into external scripts or templates
Currently the only limitation is that you cannot create compose workflows into another workflow
However using templates, script externalisation and plain code libraries you can achieve a decent code modularisation
Mike Smoot
@mes5k
Jul 20 2016 23:16
@raylim I asked the exact same questions when I started using nextflow! :) To get around the lack of sub pipelines we actually call out to nextflow from within processes. This isn't perfect, but works reasonably well. And the need to re-use processes turned out to be much less of an issue than I'd imagined once I got our pipelines properly organized. Just my two cents.