These are chat archives for nextflow-io/nextflow

3rd
Jul 2018
Evan Floden
@evanfloden
Jul 03 2018 04:48
@ShawnConecone I havn't followed the conversation but I think your question is more: Can I use execwith templates as can be done with scriptand shell?
Paolo Di Tommaso
@pditommaso
Jul 03 2018 07:11
@vkaimal try to ask for less the machine amount ag 30 MB instead of 32
KochTobi
@KochTobi
Jul 03 2018 09:22
This message was deleted
Thomas Zichner
@zichner
Jul 03 2018 13:36
Hi Paolo,
Is there a possibility to use dynamic compute resources, like memory = { 15.GB * task.attempt }, with AWS Batch?
It seems that a failed job get re-executed with the same resource requirements as in the first attempt.
Thanks
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:37
yes, it should be possible
is NF showing the task re-execution in the log ?
or is it re-executed silently ?
Thomas Zichner
@zichner
Jul 03 2018 13:38
It is re-executing silently (seems that the re-execution of failed jobs is done directly by AWS Batch rather than nextflow)
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:39
in that case no
Thomas Zichner
@zichner
Jul 03 2018 13:40
Can I somehow force Nextflow to take care of the resubmission rather than let AWS Batch do it?
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:43
as usual errorStrategy = 'retry'
Thomas Zichner
@zichner
Jul 03 2018 13:46
Yes, this I have. But my impression was that Nextflow is telling AWS Batch that the maximum number of tries is X and that AWS Batch is taking care of the rest
But maybe that is not the case
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:47
code is always the best answer .. :smile:
       if( task.config.getMaxRetries() && task.config.getErrorStrategy() != ErrorStrategy.RETRY ) {
            def retry = new RetryStrategy().withAttempts( task.config.getMaxRetries()+1 )
            result.setRetryStrategy(retry)
        }
if you set errorStrategy = 'retry' NF handles it
Thomas Zichner
@zichner
Jul 03 2018 13:51
Thanks. Just to fully understand this, does this mean that NF handles it in case the error strategy is retry or if it is not retry?
Currently, my setting is errorStrategy = { task.exitStatus in [143,137] ? 'retry' : 'finish' }
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:51
bingo
this is the beauty of open source !
no wait, now I'm getting confused as well :smile:
Thomas Zichner
@zichner
Jul 03 2018 13:52
:)
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:53
yes, this is a big quirk
Thomas Zichner
@zichner
Jul 03 2018 13:53
Sorry, don't want to confuse you ;-)
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:54
the closure is resolved to the actual value, therefore if you have an exit status 143,137 it will be retry
and NF will handle it
otherwise it the error strategy is finish
but I guess you set maxRetries therefore batch will try to re-execute it
Thomas Zichner
@zichner
Jul 03 2018 13:56
yes, I set maxRetries = 2
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:56
that is
Thomas Zichner
@zichner
Jul 03 2018 13:57
The exit code I got was 255 and I can see that AWS Batch re-executed the failed job 2 times
Paolo Di Tommaso
@pditommaso
Jul 03 2018 13:58
exactly, I would remove the dyn rule and let NF always to handle it
Thomas Zichner
@zichner
Jul 03 2018 13:59
Ok. Thank you very much. I will try this.
Maxime Vallée
@valleem
Jul 03 2018 13:59
Hello NF community : the *-all files in the releases page, the ones that are heavier, I suppose they contain the binaries needed by NF?
Paolo Di Tommaso
@pditommaso
Jul 03 2018 14:40
@valleem yes, download it, give execute permission, rename it to nextflow and you can use as drop-in replacement of the standard nextflow script
Maxime Vallée
@valleem
Jul 03 2018 14:41
thanks @pditommaso it is very useful for my off-web cluster!
Paolo Di Tommaso
@pditommaso
Jul 03 2018 14:41
it's meant exactly for that
Dave Istanto
@DaveIstanto
Jul 03 2018 16:45
@skptic Yes! thank you for clearing that up
@pditommaso quoting @skptic that helped clearing up the question I had: " Can I use exec with templates as can be done with script and shell ?"
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:17
no exec only allow to execute plan groovy/java code
Dave Istanto
@DaveIstanto
Jul 03 2018 17:18
hmm, what I meant was can I use script and the template file contains groovy/java language with: #!/usr/bin/env nextflow header?
doing it as how shell can be replaced by script with templates containing shell commands?
micans
@micans
Jul 03 2018 17:20
I have the scenario where the same pipeline can have very differently sized inputs (smart Seq and bulk RNA seq). I'd like to set two different resource settings in the config files and load one of them in a branch-type manner (e.g. distinguish between param.mode == "bulk" or "sc"). Can I worm branching somewhere into the config loading, below the level of workflow.profile?
Sorry to interrupt, I was writing that for a while
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:24
@DaveIstanto you can just use nextflow run <script> as any tool, however beware that NF will try to use all cpus for each task execution
@micans use a reverse logic, define in the config different profiles using a different set of params instead of ifs in your script
Dave Istanto
@DaveIstanto
Jul 03 2018 17:26
@pditommaso understood, thank you
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:27
:+1:
micans
@micans
Jul 03 2018 17:28
Alright ... I probably have brain-freeze at the moment. How do I trigger the loading of these different profiles? With the --profile option? Right now this loads profiles for different executor environments .. hence my question.
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:28
-profile foo (one - not two)
you can define as many profiles as you want and mix as you need .. without exaggerating .. :joy:
micans
@micans
Jul 03 2018 17:30
ah yes, sorry, I new about one -. But there is only a single argument to -profile, correct? So I will have to duplicate the profile at the top level.
(probably too tired .. almost off home now)
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:32
separate them by comma
micans
@micans
Jul 03 2018 17:32
My word, such good news before I go home. My apologies I did not see that. Thanks!
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:33
you are welcome, bring me a :beer: when you will come in Barcelona :smile:
micans
@micans
Jul 03 2018 17:34
Will definitely. It's quite a few already actually! NF is fab. Have a fab evening too!
Paolo Di Tommaso
@pditommaso
Jul 03 2018 17:35
same there !
AviatorMoser
@AviatorMoser
Jul 03 2018 19:31
Question: How would one go about canceling a workflow?
Paolo Di Tommaso
@pditommaso
Jul 03 2018 19:38
CTRL+C or kill <pid>
AviatorMoser
@AviatorMoser
Jul 03 2018 19:39
well -- it's a flow interfacing with SLURM -- apparently it's been trying to run a step on SLURM since June 13 every 5 min. The .nextflow.log reports the step was submitted-- five min later -- submitted -- five min later -- submitted -- but none of these are actually submitted to SLURM
so I would like to cancel this workflow but I do not necessarily understand how
Paolo Di Tommaso
@pditommaso
Jul 03 2018 19:41
what's the problem killing it ?
AviatorMoser
@AviatorMoser
Jul 03 2018 19:41
I don't know what to kill is the bigger issue
Paolo Di Tommaso
@pditommaso
Jul 03 2018 19:42
if you have launched in the foreground, it's enough to press CTRL+C
if you launched in the background, ps fx shows the running process and you can kill it
you you have executed as slurm job you can kill that job
you should know what your are doing ..
AviatorMoser
@AviatorMoser
Jul 03 2018 19:45
Well I don't really. We have some special setup where nextflow jobs are launched from a web server and I'm just trying to understand how it works.
Paolo Di Tommaso
@pditommaso
Jul 03 2018 19:45
ask to your sysadmin
it can be executed in many ways
AviatorMoser
@AviatorMoser
Jul 03 2018 19:48
Admin is off for the holiday. I'm learning this in real-time at the moment. It's being launched via some python code. You won't be able to help with that, so I'll dig more on my end
tbugfinder
@tbugfinder
Jul 03 2018 19:49
This raises a question in regards to stopping a foreground nextflow process with awsbatch executor using Ctrl+c. Do we expect an exception here, as usually I see such?
vkaimal
@vkaimal
Jul 03 2018 20:10
@tbugfinder @pditommaso It works now! I overrode max_memory to 40GB. By the way, the process in question was was STAR alignment, as part of @ewels 's NGI-RNAseq pipeline. Works wonderfully. Thanks @tbugfinder @pditommaso @ewels !
Jean-Christophe Houde
@jchoude
Jul 03 2018 20:11
HI all, a quick question: if a process is defined with, say, cpus 8, and I try to launch the whole flow on the local scheduler with -qs 6, what would be the expected behavior? For NF to not launch the specific process, since -qs is smaller than the required cpu count, or some other behavior?
Alexander Peltzer
@apeltzer
Jul 03 2018 20:12
@vkaimal STAR is really memory intensive
Jean-Christophe Houde
@jchoude
Jul 03 2018 20:16
Or, re-reading the doc, -qs is really on the process count, I guess?