These are chat archives for nextflow-io/nextflow

Aug 2016
Aug 03 2016 10:11

hi @pditommaso

Was playing with the error strategy + task.attempt value to see how some parallel treatment react to an error + retry.
The problem is i don't know why the retry doesn't occur
Here is my code:

  1 #!/user/bin/env nextflow
  3 source = Channel.from([0,0,0,0,0,1,0,0,0,0,0,2,0,0,0,0])
  5 process listWatcher {
  6         echo true
  8         input:
  9         val n from source
 11         output:
 12         val n into res
 14         script:
 15         """
 16         echo "n : $n"
 17         """
 18 }
 20 process exitNormalOrErrorAndRetryToCorrect{
 21         echo true
 22         errorStrategy 'retry'
 23         maxRetries 3
 25         input:
 26         val r from res
 28         script:
 29         exitVal = r - task.attempt + 1
 30         """
 31         echo "exit code : $r -> $exitVal (${task.attempt})"
 32         exit "$exitVal"
 33         """
 34 }
(it's supposed to exit 0 several time, exit 1 and then retry minus 1 -> exit 0, ...,exit 2 then retry twice in a row to finally exit 0)
if u don't mind giving a quick look ofc ;)
Paolo Di Tommaso
Aug 03 2016 12:09
@Mokok this is a bit tricky, when failing the task is re-executed at it is i.e. you wont see any change for exitVal or task.attempt values
Aug 03 2016 13:00
So, task.attempt is only available out of the "script" field ?
Paolo Di Tommaso
Aug 03 2016 13:07
we could say so
Aug 03 2016 13:10
ok, this is good to know thanks
( I admit i'm kinda looking for some of the strangest behaviors ^^ )
Aug 03 2016 14:13

Is there a way to push one process's exitStatus through a Channel ?
The purpose is to provide a following task a way to know how (one of) the previous task(s) ended ? (with both valid exit status and on-error exit status to be able to dynamically choose a process branch for example)

(I find it more logical to transmit the actual exitStatus rather than a arbitrary set variable ... if it's not over complicated)

Paolo Di Tommaso
Aug 03 2016 14:22
um, it would require some tricks
something like that
process foo {
  stdout into exitstatus

    set +e
    (set -e; your_main_script_here) > /dev/null
    echo $?
makes sense ?
Aug 03 2016 14:26
i understand is avoiding the shell to react about on-error execution (with "set -e", and the getting the status (with "$?")
am I right ?
but i dont get the "> /dev/null" part
Paolo Di Tommaso
Aug 03 2016 14:27
> /dev/null to suppress the script output
and capture only the exit status on the stdout
otherwise you will need to save the $? to a file and capture the file
Aug 03 2016 14:34

you rock ! that's one interesting solution.

but doesn't it add an high weight on each process execution and the nextFlow workflow (in terms of read/write-ability) ?

Paolo Di Tommaso
Aug 03 2016 14:48
It's very similar to what is happening behind the scene, so I don't see any particular overhead
However yes, I would suggest this for millions of fine grain tasks i.e. shorter than a sec
Aug 03 2016 15:21
did some tests, it works fine
i also tried to do it with several exit status to be collected, i just need to parse the stdout to isolate the code
and then i can let the normal cmd output be if i "wrap" the exit status to be sure to extract it with a given pattern (like <Exit: aValue> in an output looking like "foobarfoobar<Exit: *exitValue* > foobarfoo")
all this to conclude : your solution is definitively good. I thank you a lot :)
Paolo Di Tommaso
Aug 03 2016 17:51