These are chat archives for nextflow-io/nextflow

16th
Nov 2017
Luca Cozzuto
@lucacozzuto
Nov 16 2017 10:04
Hi Paolo, do you have a way to let process continue if there is only one that is failing? (and not required to the following ones)
I think at some point we will implement the support for multiple publishDir
+1 I was also asking the same improvement!
Edgar
@edgano
Nov 16 2017 10:07
Hey @lucacozzuto , maybe you are looking for the errorStrategy policy ? [Doc] (https://www.nextflow.io/docs/latest/process.html#errorstrategy)
Luca Cozzuto
@lucacozzuto
Nov 16 2017 10:15
wonderful @edgano !
Paolo Di Tommaso
@pditommaso
Nov 16 2017 10:46
:+1:
Félix C. Morency
@fmorency
Nov 16 2017 15:26
fwiw, I'm using https://launchpad.net/~jonathonf/+archive/ubuntu/openjdk as the Java8 repository for Ubuntu 14.04 Trusty since Java8 is not in the official repositories
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:25
;)
Félix C. Morency
@fmorency
Nov 16 2017 16:36
hey @pditommaso I'm having a wierd behaviour in my pipeline. I have something like
foo = true
bar.ifEmpty {
  foo = false
  log.info "$foo"
}

log.info "$foo"
if(!foo) {
  process A {}
} else {
  process B {}
}
the print from the ifEmpty gives false but the other print gives true
not sure I understand what's going on
it was working with the local executor but fails to execute properly using slurm
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:39
don't forget operators are asynchronous, ifEmpty may ne executed after
log.info "$foo"
if(!foo) {
  process A {}
} else {
  process B {}
}
Félix C. Morency
@fmorency
Nov 16 2017 16:40
Ah. Is there a synchronous way of checking is a channel is empty?
I want to enable/disable certain branches of the pipeline depending on the input files I detect
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:42
there's fallacy of composition
in the meaning you should refactor your code to not need that
Félix C. Morency
@fmorency
Nov 16 2017 16:43
mind to shed some light on this? not sure I understand how to do that
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:44
so basically you want process A to be execute only if there's a value in a channel, otherwise B ?
Félix C. Morency
@fmorency
Nov 16 2017 16:46
yes
(that's a simplistic toy example, but yes)
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:47
ok the first part is nothing special, is just the default process behaviour as long it has the channel declared as input, right?
Félix C. Morency
@fmorency
Nov 16 2017 16:47
yes
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:48
so, the problem is the otherwise part
you should be able to do something like
Félix C. Morency
@fmorency
Nov 16 2017 16:49
that and the fact that both A and B output "in the same channel" at the end of the {} scope.
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:52
channel_b = Channel.create()
channel_main.tap { channel_a } .ifEmpty { channel_b << 'WHATEVER' }

process A {
  input:
  val x from channel_a

}

process B {
  input:
  val x from channel_b

}
mmm, no
Félix C. Morency
@fmorency
Nov 16 2017 16:54
And how would you solve the issue that a process C takes input from either process A or process B?
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:55
that's easy
out_a.mix(out_b).set { channel_C }
Félix C. Morency
@fmorency
Nov 16 2017 16:55
yes ok
Paolo Di Tommaso
@pditommaso
Nov 16 2017 16:56
but there's still a problem, the first hang because channel_b is never closed, when there's some data
ok, maybe the easiest solution is
channel_main.count().set{ channel_counter }

process A {
  input:
  val x from channel_counter
  when:
  x  > 0 
}

process B {
  input:
  val x from channel_counter
  when:
  x  ==  0 

}
this should work
Félix C. Morency
@fmorency
Nov 16 2017 17:00
Mmm ok, I will investigate this. It looks a bit like a hack tho
ie. overly complex/non-intuitive for such a simple behaviour
Paolo Di Tommaso
@pditommaso
Nov 16 2017 17:03
why!? I like the symmetry !
Félix C. Morency
@fmorency
Nov 16 2017 17:04
I mean, the main issue can be formulated as if input_foo execute process A else execute process B. process C will take either outputs as it's input
Paolo Di Tommaso
@pditommaso
Nov 16 2017 17:05
well, NF has not yet Siri like ability, but you can implemented this pattern ;)
Félix C. Morency
@fmorency
Nov 16 2017 17:06
Haha I see. Thanks for the workaround, I will investigate this
Paolo Di Tommaso
@pditommaso
Nov 16 2017 17:06
feel free to propose an improvement
Félix C. Morency
@fmorency
Nov 16 2017 17:07
I will think about it and create an issue so other can discuss
Félix C. Morency
@fmorency
Nov 16 2017 18:33
@pditommaso made the change in my pipeline and it seemed to have worked first try. Thanks!
Paolo Di Tommaso
@pditommaso
Nov 16 2017 18:36
:v:
Félix C. Morency
@fmorency
Nov 16 2017 18:39
still no donation button on the NF web-page btw ;)
Paolo Di Tommaso
@pditommaso
Nov 16 2017 18:41
it will happen in due time ;)
Félix C. Morency
@fmorency
Nov 16 2017 18:42
any plans on going commercial?
Paolo Di Tommaso
@pditommaso
Nov 16 2017 18:43
can't confirm nor deny
Félix C. Morency
@fmorency
Nov 16 2017 18:43
keep me posted - I'm interested.
Paolo Di Tommaso
@pditommaso
Nov 16 2017 18:43
:ok_hand:
Mike Smoot
@mes5k
Nov 16 2017 18:57
you're gonna make me pay for modules, aren't you? :)
Paolo Di Tommaso
@pditommaso
Nov 16 2017 18:57
nooo
Félix C. Morency
@fmorency
Nov 16 2017 18:59
NF free for local executor only. Executors are all DLC ;)
Paolo Di Tommaso
@pditommaso
Nov 16 2017 19:00
local executor 1 cpus :)
Mike Smoot
@mes5k
Nov 16 2017 19:00
Does that mean you'll debug my slurm queue then? :)
Paolo Di Tommaso
@pditommaso
Nov 16 2017 19:00
$1000 for each extra cpus :joy:
have found the problem with slurm ?
Mike Smoot
@mes5k
Nov 16 2017 19:03
Well, I think I found it, but I haven't figured out what to do about it yet. Basically, when slurm restarts to add new nodes, some tasks appear to get dropped. This has only been a problem for us with one pipeline that generates thousands of small jobs. I'm running an older version of slurm and plan to upgrade, but I haven't found the time yet.
Paolo Di Tommaso
@pditommaso
Nov 16 2017 19:03
I see
ugly
Mike Smoot
@mes5k
Nov 16 2017 19:04
yup