These are chat archives for nextflow-io/nextflow

16th
Feb 2017
LukeGoodsell
@LukeGoodsell
Feb 16 2017 09:47 UTC

Good morning. Is there any way to selectively run portions of a Nextflow script? I’d like to be able to prepare unit tests for particular portions, but the only solution I can see is to use nested sub-workflows.

#238 Will probably address this, too, but is there any other approach I’ve overlooked?

Maxime Garcia
@MaxUlysse
Feb 16 2017 09:49 UTC
@LukeGoodsell Look into https://github.com/nextflow-io/awesome-nextflow there are several pipelines that have steps possiblities
Paolo Di Tommaso
@pditommaso
Feb 16 2017 09:49 UTC
my suggestions are to externalise process logic with script or templates
to test the overall pipeline logic, create a Docker image and sample dataset that allows you to run your pipeline in a few minutes and test it Travis or CircleCI eg. https://circleci.com/gh/nextflow-io/rnatoy
LukeGoodsell
@LukeGoodsell
Feb 16 2017 09:56 UTC
Thanks, Paolo. I’m happy with dockerising and testing (via CI or otherwise) the whole pipeline. With the template approach, each process (that I wish to unit-test) must be written in an external file, right? (Not a criticisim, just a clarification)
Paolo Di Tommaso
@pditommaso
Feb 16 2017 09:57 UTC
yes, than each template can be unit tested as a bash script
LukeGoodsell
@LukeGoodsell
Feb 16 2017 09:57 UTC
OK. Thanks.
Paolo Di Tommaso
@pditommaso
Feb 16 2017 09:58 UTC
but IMO testing the pipeline with the CI server is more than enough
LukeGoodsell
@LukeGoodsell
Feb 16 2017 10:00 UTC
My issue with that is just that I’m developing a big (think multi-week-run) pipeline and am considering using Nextflow. To test the whole thing for all circumstances is not feasible, so I wish to test each unit for all circumstances and test the whole workflow for just selected cases.
Paolo Di Tommaso
@pditommaso
Feb 16 2017 10:01 UTC
I see, makes sense
Anthony Underwood
@aunderwo
Feb 16 2017 11:31 UTC
I am getting an error when trying to run rnatoy with singularity
./nextflow run rnatoy -with-singularity
N E X T F L O W  ~  version 0.23.3
Pulling nextflow-io/rnatoy ...
 downloaded from https://github.com/nextflow-io/rnatoy.git
Launching `nextflow-io/rnatoy` [adoring_joliot] - revision: 1495129c1d [master]
R N A T O Y   P I P E L I N E
=================================
genome             : /home/vagrant/.nextflow/assets/nextflow-io/rnatoy/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
annotat            : /home/vagrant/.nextflow/assets/nextflow-io/rnatoy/data/ggal/ggal_1_48850000_49020000.bed.gff
reads              : /home/vagrant/.nextflow/assets/nextflow-io/rnatoy/data/ggal/*_{1,2}.fq
outdir             : results
[warm up] executor > local
[d2/7d3853] Submitted process > buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)
ERROR ~ Error executing process > 'buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)'

Caused by:
  Process `buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank)` terminated with an error exit status (255)

Command executed:

  bowtie2-build --threads 1 ggal_1_48850000_49020000.Ggal71.500bpflank.fa genome.index

Command exit status:
  255

Command output:
  (empty)

Command error:
  ERROR  : Illegal input character '@' in: 'SINGULARITY_IMAGE=/home/vagrant/singularity/nextflow/rnatoy@sha256:9ac0345b5851b2b20913cb4e6d469df77cf1232bafcadf8fd929535614a85c75'
  ABORT  : Retval = 255

Work dir:
  /home/vagrant/singularity/work/d2/7d38535197c15ed6f937c167c718b8

Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'

 -- Check '.nextflow.log' file for details
Oops .. something went wrong
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:48 UTC
Command error:
  ERROR  : Illegal input character '@' in: 'SINGULARITY_IMAGE=/home/vagrant/singularity/nextflow/rnatoy@sha256:9ac0345b5851b2b20913cb4e6d469df77cf1232bafcadf8fd929535614a85c75'
  ABORT  : Retval = 255
Anthony Underwood
@aunderwo
Feb 16 2017 11:50 UTC
Hi. Yes I saw that but I thought since I was running a vanilla command nextflow run rnatoy -with-singularity this must be a bug
I'm not specifying an image but nextflow is creating this I believe
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:51 UTC
The problem is that Singularity does not support the Docker image name which uses the digest ID
Anthony Underwood
@aunderwo
Feb 16 2017 11:52 UTC
OK so for the example rnatoy this isn't singularity compatible?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:52 UTC
well, the real problem is that though Singularity can download Docker images from the Docker hub
you will need to create a Singularity image in your local file system and specify that in the NF config file
Anthony Underwood
@aunderwo
Feb 16 2017 11:54 UTC
OK so shall i do this manually?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:54 UTC
yep
one way is to this tool
Anthony Underwood
@aunderwo
Feb 16 2017 11:56 UTC
OK download the docker image and then convert to a singularity image
?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:56 UTC
that will create a singularity image file which you need to specify in the nextflow.config file
yes
Anthony Underwood
@aunderwo
Feb 16 2017 11:56 UTC
Where do I find the docker image for rnatoy?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:57 UTC
here
oops
Anthony Underwood
@aunderwo
Feb 16 2017 11:58 UTC
OK that's on dockerhub?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:58 UTC
yes
the image name is nextflow/rnatoy@sha256:9ac0345b5851b2b20913cb4e6d469df77cf1232bafcadf8fd929535614a85c75
Anthony Underwood
@aunderwo
Feb 16 2017 11:58 UTC
Thanks. I'll give that a go :)
Paolo Di Tommaso
@pditommaso
Feb 16 2017 11:58 UTC
cool
also I've noticed that you are using a vagrant box
Anthony Underwood
@aunderwo
Feb 16 2017 12:00 UTC
yes because I am trying singularity on a mac!
Paolo Di Tommaso
@pditommaso
Feb 16 2017 12:00 UTC
of course
but make sure to not run it in a mounted/shared folder
Anthony Underwood
@aunderwo
Feb 16 2017 12:01 UTC
ok just run in /home/vagrant for instance?
Paolo Di Tommaso
@pditommaso
Feb 16 2017 12:01 UTC
should be fine
Anthony Underwood
@aunderwo
Feb 16 2017 12:01 UTC
OK thanks
Paolo Di Tommaso
@pditommaso
Feb 16 2017 12:01 UTC
welcome
Anthony Underwood
@aunderwo
Feb 16 2017 13:39 UTC
@pditommaso That worked a treat thanks
LukeGoodsell
@LukeGoodsell
Feb 16 2017 13:42 UTC

Hi again. If I create two channels from pre-defined variables, e.g.:

zeroToNine = Channel.from( 0..9 )
strings = Channel.from( 'A'..'Z’ )

… and then take those two channels as input to a process, e.g.:

process test {
    input:
    val num from zeroToNine
    val string from strings

    ...

}

... can I be certain that the same pairings will happen each time ( [0, ‘A’], [1, ‘B’] …)?

I’ve run some tests and it seems to be the case, but I just want to be sure.
Paolo Di Tommaso
@pditommaso
Feb 16 2017 13:51 UTC
Yes, it will. A channel is an asynchronous FIFO structure, thus the ordering is guaranteed
note however that your example will trigger 10 runs of test having [0, 'J'] as the last pair
if I've counted well :)
LukeGoodsell
@LukeGoodsell
Feb 16 2017 13:54 UTC
but there isn’t a chance that process N will take from num, then process N+1 takes from num, then from string, and then process N takes from N again?
e.g.:
Process test (1) takes ‘0’ from num
Process test (2) takes ‘1’ from num
Process test (2) takes ‘A’ from strings
Process test (1) takes ‘B’ from stirngs
Paolo Di Tommaso
@pditommaso
Feb 16 2017 13:54 UTC
nope
LukeGoodsell
@LukeGoodsell
Feb 16 2017 13:55 UTC
Great. Thanks, as always :+1:
Paolo Di Tommaso
@pditommaso
Feb 16 2017 13:55 UTC
that's how dataflow works
an execution is trigger only when the all expected inputs are available
LukeGoodsell
@LukeGoodsell
Feb 16 2017 13:56 UTC
Ah, I see. Good to know
Paolo Di Tommaso
@pditommaso
Feb 16 2017 13:56 UTC
ie. in your case a num and a string otherwise it waits
in other words it manages all the race conditions for you