These are chat archives for nextflow-io/nextflow

17th
Oct 2016
Félix C. Morency
@fmorency
Oct 17 2016 13:31
morning
Félix C. Morency
@fmorency
Oct 17 2016 13:36
mmm I updated to latest version and have a No project name was specified now
it doesn't like the nextflow run -with-dag dag.png -with-timeline project.nf
Félix C. Morency
@fmorency
Oct 17 2016 13:44
i understand the problem, but it was working with previous version (0.22.0). regression?
Mokok
@Mokok
Oct 17 2016 13:45
i personally put it like "./nextflow [nextflowOpt] run <myscript.nf> [scriptOpt]", don't know if the order rly matters, have u tried different params order ?
Félix C. Morency
@fmorency
Oct 17 2016 13:46
Mokok, yeah, it works with -with-timeline -with-dag dag.png project.nf
Mokok
@Mokok
Oct 17 2016 13:46
ho, nvm then, i go back in my lair ^^
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:48
@fmorency -with-timeline takes an optional argument that is the time file name, thus
nextflow run -with-dag dag.png -with-timeline project
project is interpreted as the timeline file name resulting in that error message
Félix C. Morency
@fmorency
Oct 17 2016 14:49
@pditommaso yes, i know, but it was working with 0.22.0 ;)
it's okay. non-issue to me
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:50
looks weird, it shouldn't
ok
Félix C. Morency
@fmorency
Oct 17 2016 14:51
what's the proper way of restarting the pipeline from a specific task?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:51
there's no way
if previous tasks have been already computed they will be automatically skipped
Félix C. Morency
@fmorency
Oct 17 2016 14:53
will it work if I delete the outputs?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:53
nope
well, depends what you mean for outputs
cached results are kept in the work dir
Félix C. Morency
@fmorency
Oct 17 2016 14:54
stuff in publishDir
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:54
yes, you can delete that
does not affect the caching
Félix C. Morency
@fmorency
Oct 17 2016 14:55
so the task will not be re-run?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:55
no
Félix C. Morency
@fmorency
Oct 17 2016 14:55
ok. do you have plans to support this?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:56
this what? restarting from a specific task ?
Félix C. Morency
@fmorency
Oct 17 2016 14:56
yes. and/or cleaning the cache for a specific task and resume the pipeline from there
the only way I found was to modify the code of the task in the .nf file
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:57
does not make much sense in the dataflow model on which NF is based
if task B depends on A, the latter needs to be executed
Félix C. Morency
@fmorency
Oct 17 2016 14:58
correct
"cleaning" A implies cleaning B also
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:58
no necessary,
Félix C. Morency
@fmorency
Oct 17 2016 14:59
This message was deleted
Paolo Di Tommaso
@pditommaso
Oct 17 2016 14:59
if A will produce an identical output, B may be skipped
Félix C. Morency
@fmorency
Oct 17 2016 14:59
yes correct
what if you change something in the script called by process A?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 15:01
it must be said that cache id a determined by using file timestamps
what if you change something in the script called by process A?
since it's a different process it will be re-executed
Félix C. Morency
@fmorency
Oct 17 2016 15:03
what if you do
""" some_script.py """
Paolo Di Tommaso
@pditommaso
Oct 17 2016 15:04
.. and you change some_script.py ..
no magic here
Félix C. Morency
@fmorency
Oct 17 2016 15:04
correct
Paolo Di Tommaso
@pditommaso
Oct 17 2016 15:04
NF don't know about that
Félix C. Morency
@fmorency
Oct 17 2016 15:05
ok. in that case, doesn't it make sense to force re-execution manually?
Paolo Di Tommaso
@pditommaso
Oct 17 2016 15:05
though templates should support that
doesn't it make sense to force re-execution manually?
um, yes
just delete that task work dir
Félix C. Morency
@fmorency
Oct 17 2016 15:08
make sense
fyi i presented NF to my team today
they're all on-board so far
Paolo Di Tommaso
@pditommaso
Oct 17 2016 15:23
cool !
amacbride
@amacbride
Oct 17 2016 21:20
I have a situation where I want to run an external script over an input file to calculate a value, then pass that value on to subsequent steps in my NF script. What do you all think would be the most straightforward way to do this? I was thinking about using an exec block and Groovy execute(), but I wasn't sure.
Paolo Di Tommaso
@pditommaso
Oct 17 2016 21:38
why using a groovy execute instead of a NF process ?
amacbride
@amacbride
Oct 17 2016 21:42
This is within a NF process. I's like to get the value from an external script, and then pass it to following pipeline steps.
It's not clear to me how I would do that -- essentially, grad the stdout of an external process and parse it into a NextFlow variable that I could pass along.
Paolo Di Tommaso
@pditommaso
Oct 17 2016 21:44
still not understanding, could u provide an example?
amacbride
@amacbride
Oct 17 2016 21:47

Suppose a I have a script, call it externalScript.pl, that does a calculation on something (in this case, the number of unmapped reads in a sequencing run). It prints the value "123456" to stdout.

I'd like a NF process that runs the script, captures the value, then can pass "12345" to a subsequent NF process, so that I can use that value as part of a downsampling step.

Paolo Di Tommaso
@pditommaso
Oct 17 2016 21:47
what about
process foo {
  output: stdout into my_channel
  '''
  externalScript.pl
  ''' 
}