These are chat archives for nextflow-io/nextflow

16th
Mar 2017
Fredrik Boulund
@boulund
Mar 16 2017 10:43
Hi!
Does Nextflow support SLURM's feature for specifying dependent jobs? (i.e. submitting all jobs at once, specifying which downstream jobs depend on some upstream ones). I'm guessing this potentially could improve the scheduler's efficiency in planning jobs. It doesn't look like it's using this feature, which is why I'm asking. :)
Paolo Di Tommaso
@pditommaso
Mar 16 2017 11:48
Nope, we are not supporting it. Mainly because tasks scheduling is computed at runtime, thus it's not possible to pre-determine the full graph of tasks dependencies
Fredrik Boulund
@boulund
Mar 16 2017 11:52
Ah, I see! No worries, just wondering :)
Fredrik Boulund
@boulund
Mar 16 2017 12:14
Is there a way to have Nextflow continue with independent tasks if there's an error for one task? Like the --keep-going option in Snakemake...
Fredrik Boulund
@boulund
Mar 16 2017 12:58
I'm guessing I could have an errorStrategy to be retry for e.g. three times, with increasing resources, and then change the error strategy to ignore if it fails a fourth?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:22
@boulund Yes, errorStrategy = 'finish'
Fredrik Boulund
@boulund
Mar 16 2017 13:23
@pditommaso Wow... how could I not see that? Sorry
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:23
what do you mean ?
Fredrik Boulund
@boulund
Mar 16 2017 13:24
I've read that part of the docs many times but I just can't recall ever reading about finish :D
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:27
:smile:
Fredrik Boulund
@boulund
Mar 16 2017 13:30
can I combine it with some kind of logic like I mentioned before, having retry first, then changing to finish after maxErrors?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:31
yes, you need to express that with a closure
eg:
errorStrategy { task.exitStatus == 140 ? 'retry' : 'terminate' }
Fredrik Boulund
@boulund
Mar 16 2017 13:32
so that would make it retry if exitStatus is 140, until maxErrors, then terminate?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:33
.. otherwise terminate
I mean
it makes it to retry if exitStatus is 140, until maxErrors, otherwise terminate
Fredrik Boulund
@boulund
Mar 16 2017 13:34
ok, then we mean the same thing. Good. I'm slightly confused by these ternary operators
Thank you!
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:35
if's so beautiful! how pythonists can live without it :)
Fredrik Boulund
@boulund
Mar 16 2017 13:35
;) we make do
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:35
it's just a shortcut for
if( task.exitStatus == 140 ) then return 'retry' else return 'terminate'
Fredrik Boulund
@boulund
Mar 16 2017 13:36
it's expressed like <something> if <condition> else <something else> in Python
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:37
I see
Fredrik Boulund
@boulund
Mar 16 2017 13:37
They are very similar, just slightly different. Enough to really mess with my head
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:38
so true
Fredrik Boulund
@boulund
Mar 16 2017 13:39

but.. I just noticed I already had

errorStrategy = {task.exitStatus == 143 ? 'retry' : 'finish'}
maxRetries = 2

specified in my config file. It still terminates as soon as one process fails two times. What have I done wrong?

Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:39
they must be defined in the process scope
process {
  errorStrategy = {task.exitStatus == 143 ? 'retry' : 'finish'}
  maxRetries = 2
}
Fredrik Boulund
@boulund
Mar 16 2017 13:40
ah yes, but they are. Sorry for not including the overall scope
process {                                                        
    executor = 'slurm'                                           
    clusterOptions = {                                           
        "-A ${params.project}" + (params.clusterOptions ?: '')   
    }                                                            
    errorStrategy = {task.exitStatus == 143 ? 'retry' : 'finish'}
    maxRetries = 2                                               
    scratch = true                                               
    stageInMode = 'copy'                                         
    stageOutMode = 'copy'                                        

    // Resource and module requirements for processes            
 etc....
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:42
thinking
how is the output produced
Fredrik Boulund
@boulund
Mar 16 2017 13:45
how do you mean? I could paste the entire stdout output from the run if that could help
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:45
the NF output
Fredrik Boulund
@boulund
Mar 16 2017 13:45
yep
(it's fairly long)
Paolo Di Tommaso
@pditommaso
Mar 16 2017 13:46
copy it to pastebin.com if possible
Paolo Di Tommaso
@pditommaso
Mar 16 2017 14:12
it's a bit more tricky
    errorStrategy = { task.exitStatus == 143 && task.attempt<3 ? 'retry' : 'finish'}
    maxRetries = 2
Fredrik Boulund
@boulund
Mar 16 2017 14:50
Ah! That's super awesome. Thank you. Do you think that's what caused it to terminate in my case?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 14:50
yes
Fredrik Boulund
@boulund
Mar 16 2017 14:50
the more I look at it, there more it seems that the process that made the workflow terminate wasn't actually re-run?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 14:51
I don't know..
Fredrik Boulund
@boulund
Mar 16 2017 15:46
But that is super helpful! I'll see if it fixes my issues.
Thank you again!
Paolo Di Tommaso
@pditommaso
Mar 16 2017 15:46
welcome!
Karin Lagesen
@karinlag
Mar 16 2017 18:33
hi
I am finally getting round to trying nextflow on the cluster
but, I am getting a memory error:
Exception in thread "main" java.lang.UnsatisfiedLinkError: /cluster/software/VERSIONS/java/jdk1.7.0_80/jre/lib/amd64/libnio.so: /cluster/software/VERSIONS/java/jdk1.7.0_80/jre/lib/amd64/libnio.so: failed to map segment from shared object: Cannot allocate memory
the cluster has limits on how much memory I can use on a login node, and I suspect that is the problem
(slurm cluster btw)
question is, is there any way of reducing this so that I can actually take it for a spin)
?
Paolo Di Tommaso
@pditommaso
Mar 16 2017 19:40
You can set the mem limits as shown below
export NXF_OPTS='-Xms512M -Xmx4G'
where -Xms is the min heap size and -Xmx is the max heap size
Karin Lagesen
@karinlag
Mar 16 2017 20:13
awesome, thanks!
Paolo Di Tommaso
@pditommaso
Mar 16 2017 20:13
:+1: