These are chat archives for nextflow-io/nextflow

8th
Jan 2018
Félix C. Morency
@fmorency
Jan 08 2018 14:45
Hi!
Is there a way to combine both retry and finalize error strategies? I would like each process to be retried at least 3 times and finalize the pipeline otherwise
Edgar
@edgano
Jan 08 2018 14:51
using maxRetries = 3 in the config file
you will retry the execution 3 times and then the execution will finish
Félix C. Morency
@fmorency
Jan 08 2018 14:52
IIRC, that will not allow the pipeline to finalize. It will stop the execution of the whole pipeline if one of the process fails 3 times
Edgar
@edgano
Jan 08 2018 14:54
a ok. my fault. I didnt get your idea.
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:17
yes, use dynamic error strategy eg
errorStrategy { task.attempt<=3 ? 'retry' : 'finish' }
maybe we could add a shortcut as retry_and_finish ..
Félix C. Morency
@fmorency
Jan 08 2018 15:19
is the retry per process or will the error strategy change to finish as soon as one process fails 3 times?
ie. all subsequent processes will be run only 1 time until finalize
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:22
it depends how subsequent processes are scheduled I guess
Félix C. Morency
@fmorency
Jan 08 2018 15:26
With the line you pasted, what will happen to processes that are scheduled after one process fails 3 times?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:28
a process can spawn different tasks in parallel
so, do you mean if "process fails 3 times" or "(the same) task fails 3 time"
Félix C. Morency
@fmorency
Jan 08 2018 15:29
the latter
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:30
it should stop after all running tasks complete
Félix C. Morency
@fmorency
Jan 08 2018 15:30
process Foo {
}
I want Foo to be retried 3 times
And after that, I want the pipeline to be able to finish (ie. do not terminate on the first error encoutered)
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:32
if an Foo task errored 4 time the overall pipeline stop
Félix C. Morency
@fmorency
Jan 08 2018 15:32
Yes that is the basic retry errorStrategy that I'm currently using
Now let's say I have
data = Channel.from(1, 2, 3)
process Foo {
   input:
   val i from data

   output:
   val o into for_bar

    ...
}

process Bar {
   input:
   val i from for_bar
   ...
}
And suppose Foo fails for the value 1
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:38
then?
Félix C. Morency
@fmorency
Jan 08 2018 15:38
I would like to
  • Retry Foo 3 times for the value 1. At the fourth time, ignore the error and continue the pipeline until the end and report the error then
  • Execute Foo for values 2 and 3
  • Execute Bar for values 2 and 3
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:39
so, use ignore instead of finish
Félix C. Morency
@fmorency
Jan 08 2018 15:40
Now let's say that Foo for value 1 has been executed 4 times. What happen if Bar fails for value 2?
Will it be retried?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:41
if you use finish it stop, with ignore it's retried
Félix C. Morency
@fmorency
Jan 08 2018 15:42
And with ignore, will the errors be reported at the end of the pipeline execution?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:43
it prints a message during the pipeline execution
Félix C. Morency
@fmorency
Jan 08 2018 15:44
Our execution log can be quite big. Is there a way to pinpoint those at the end of the execution?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:44
it's supposed to be the goal of this issue
nextflow-io/nextflow#534
Félix C. Morency
@fmorency
Jan 08 2018 15:46
Cool! Thanks @pditommaso
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:46
:+1:
Félix C. Morency
@fmorency
Jan 08 2018 15:57
For some reason, it doesn't like the task.attempt <= 3
Cannot compare groovy.util.ConfigObject with value '[:]' and java.lang.Integer with value '3'
Paolo Di Tommaso
@pditommaso
Jan 08 2018 15:58
Provide the full snippet
Félix C. Morency
@fmorency
Jan 08 2018 15:59
process {
    errorStrategy { task.attempt <= 3 ? 'retry' : 'ignore' }
    maxRetries = 3
    maxErrors = -1
}
Paolo Di Tommaso
@pditommaso
Jan 08 2018 16:01
process foo {
    errorStrategy { task.attempt <= 3 ? 'retry' : 'ignore' }
    maxRetries 3

  '''
  exit 1
  '''
}
$ nextflow run test.nf
N E X T F L O W  ~  version 0.26.4
Launching `t` [gloomy_mcclintock] - revision: 6cd9d91138
[warm up] executor > local
[a4/f2ef55] Submitted process > foo
[a4/f2ef55] NOTE: Process `foo` terminated with an error exit status (1) -- Execution is retried (1)
[29/1f29c1] Re-submitted process > foo
[29/1f29c1] NOTE: Process `foo` terminated with an error exit status (1) -- Execution is retried (2)
[1d/c18df4] Re-submitted process > foo
[1d/c18df4] NOTE: Process `foo` terminated with an error exit status (1) -- Execution is retried (3)
[2d/5ffe88] Re-submitted process > foo
[2d/5ffe88] NOTE: Process `foo` terminated with an error exit status (1) -- Error is ignored
Félix C. Morency
@fmorency
Jan 08 2018 16:03
Does it work if you place the line in the nextflow.config?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 16:05
yes
Félix C. Morency
@fmorency
Jan 08 2018 16:07
It works if a place the line in the main.nf, but not in the nextflow.config
Paolo Di Tommaso
@pditommaso
Jan 08 2018 16:07
umm
in the config, it must be
   process.errorStrategy = { task.attempt <= 3 ? 'retry' : 'ignore' }
   process.maxRetries = 3
Félix C. Morency
@fmorency
Jan 08 2018 16:08
Well
Paolo Di Tommaso
@pditommaso
Jan 08 2018 16:08
in the config you need =, in the script you don't need it
Félix C. Morency
@fmorency
Jan 08 2018 16:08
Ok, I was missing a =
Félix C. Morency
@fmorency
Jan 08 2018 16:15
Good, poc works. Thanks!
Paolo Di Tommaso
@pditommaso
Jan 08 2018 16:20
:v:
Félix C. Morency
@fmorency
Jan 08 2018 16:41
@pditommaso just to make sure, at the moment, the WHY of ignored error aren't reported anywhere. Will 534 address this?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:07
They ARE reported
it's printed in the stdout
Félix C. Morency
@fmorency
Jan 08 2018 18:07
Mmm I don't see the cause anywhere
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:07
and they are reported as failed tasks in the trace file
[37/05f856] NOTE: Process `foo` terminated with an error exit status (1) -- Execution is retried (1)
Félix C. Morency
@fmorency
Jan 08 2018 18:08
Yes I see those lines, but I don't see the Caused by: nor the Command executed: or Command error: blocks
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:09
because it supposed that task is failed for an expected reason
hence is only reported that exit status
you can still look in task work dir and troubleshoot the problem
also nextflow log <filter> will help on that
Félix C. Morency
@fmorency
Jan 08 2018 18:20
I see. Would be nice to have that info directly available
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:21
it would be unreadable as soon as you have more than one error .. (most common situation)
Félix C. Morency
@fmorency
Jan 08 2018 18:22
Agreed. Is there a nextflow log filter to get the Command error:?
Or it's just never reported?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:27
nextflow log last -filter  "exit == 1 " -f stdout
Félix C. Morency
@fmorency
Jan 08 2018 18:28
Mmm it only prints a -
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:28
stderr ?
Félix C. Morency
@fmorency
Jan 08 2018 18:28
but the .command.err contains the error
stderr prints a bunch of true
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:29
so .command.log is your friend :)
Félix C. Morency
@fmorency
Jan 08 2018 18:29
I meant .command.err
Ok, I can hack my way ;)
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:30
weird if so
Félix C. Morency
@fmorency
Jan 08 2018 18:30
.command.err contains the Traceback of the python error
Which is what I'm looking for and what is being reported when a pipeline fails
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:31
ummm, I think we have a bug here
it seems it prints trueinstead of the actual error message
Félix C. Morency
@fmorency
Jan 08 2018 18:32
You are able to reproduce?
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:32
so it seems
you may want to open an issue for that
Félix C. Morency
@fmorency
Jan 08 2018 18:33
Will do. Thanks a lot :)
Paolo Di Tommaso
@pditommaso
Jan 08 2018 18:33
you are welcome