These are chat archives for nextflow-io/nextflow

15th
Jun 2016
Julian Mazzitelli
@thejmazz
Jun 15 2016 02:38
is there any way to do a dry run? or becuase dataflow is "push" you cant?
Paolo Di Tommaso
@pditommaso
Jun 15 2016 05:12
@thejmazz dry run is not possible in some extent for the "push" model as you are suggesting, and also because process outputs are resolved at runtime, so there's not way to implement a consistent dry-run feature
the best practice is to include in your pipeline a dataset (even a fake one) small enough to to run your pipeline in a few seconds
Julian Mazzitelli
@thejmazz
Jun 15 2016 05:15

I thought as much. with nextflow much easier than snakemake to design "branching" pipelines. snakemake's wildcarding regex becomes too ambiguous and I was not able to

  1. call without trimming (raw fastq-dump)
  2. call with trimming

wherease with nextflow, it is easy to create branches using

( myChannel1, ..., myChanneln ) = myChannel.into(n)

But snakemake has nice dry run :p

Paolo Di Tommaso
@pditommaso
Jun 15 2016 05:16
there's no such perfect tool :)
Julian Mazzitelli
@thejmazz
Jun 15 2016 05:17
hehe yes. I will try to put what I like from both into the bionode one. and it will have its own faults as well :D
Paolo Di Tommaso
@pditommaso
Jun 15 2016 05:18
the good of the streaming approach is that scale much better when compared to snakemake
becuase nxf doesn't need to create to complete execution DAG to run it
thus it needs much less resources for git computation
also you can resolve dependencies at runtime, which is not possible using a "static" approach
Julian Mazzitelli
@thejmazz
Jun 15 2016 05:28
by dependencies do you mean the input and ouput of each process?
Paolo Di Tommaso
@pditommaso
Jun 15 2016 05:28
@huguesfontenelle if you want test the retry mechanism I suggest you to add an exit 1 at the end of your script instead. Or maybe something like [[ ! -f $HOME/dummy ]] && touch $HOME/dummy && exit 1
@thejmazz well no, I mean that you can dynamic branchs in your workflow
in the meaning that a branch is evaluated a runtime
see for example this post by @samuell
(need to go)
Julian Mazzitelli
@thejmazz
Jun 15 2016 05:35
nice post. yep I've read his make one before
Julian Mazzitelli
@thejmazz
Jun 15 2016 06:21
trying nxf on server, cannot touch '.command.trace', any suggestions? installed java with sudo apt-get install default-jre
I own the dir
Screen Shot 2016-06-15 at 2.18.01 AM.png
Julian Mazzitelli
@thejmazz
Jun 15 2016 06:31
the hello world pipeline works fine (and creates .command.trace), it must be something with my pipeline or setup
Paolo Di Tommaso
@pditommaso
Jun 15 2016 07:30
@thejmazz Quite weird, well it's surely some problem with access permissions. Are u using docker ?
Julian Mazzitelli
@thejmazz
Jun 15 2016 13:10
I have containers for each image yes, but nxf is running "on the metal"'. I'm gonna try running nextflow from inside a container, and then I can turn off the process containers
Paolo Di Tommaso
@pditommaso
Jun 15 2016 13:11
that means that the job run in a container, right?
I would try to move in the task directory and debug a bit that tasks
something strange with file permissions
try to run bash -x .command.run
Mike Smoot
@mes5k
Jun 15 2016 21:20
@pditommaso Hi Paolo, I'm using splitFasta on a multifasta file and I'm getting this error Parameter `file` requires a split size grater than 1 for operator: splitFasta. Can you explain the rationale for this restriction and if there's an easy way to get around it? I'd like each record in the multifasta file to be in its own file.
Paolo Di Tommaso
@pditommaso
Jun 15 2016 21:21
can I see the code snippet ?
Mike Smoot
@mes5k
Jun 15 2016 21:21
fasta_scaffolds.splitFasta( file: true )
I see the check at line 76 of AbstractTextSplitter.groovy, I'm just not sure why it's there.
And this is what generated fasta_scaffolds: output: file("scaffolds.fa") into fasta_scaffolds
Paolo Di Tommaso
@pditommaso
Jun 15 2016 21:23
I think it's a bug
could you please open an issue
Mike Smoot
@mes5k
Jun 15 2016 21:25
Sure, will do
Mike Smoot
@mes5k
Jun 15 2016 21:33
I see how I can use the file parameter in splitFasta to give the file a name, but I'm wondering if it would be possible to use the fasta record id to create a file?
Paolo Di Tommaso
@pditommaso
Jun 15 2016 21:43
no out of the box, but I guess you can do that chaining a split and a map
for example:
yourFile.splitFasta().map {  def file = saveToFile(it); return file }
where saveToFile is pseudocode
Mike Smoot
@mes5k
Jun 15 2016 21:55
Yup, that's where I was headed - thanks!