process.errorStrategy = 'retry'
process.maxRetries = 5 // or more
Hello to the NextFlow Community! What is the best way to trouble shoot an issue with trying to launch a nf workflow from tower using AWS Batch. My workflow worked great until recently when I tried to rerun it. There is likely an issue on the AWS side with my Jobs stuck in runnable status. I have exhausted all of my resources and looked at every option to resolve this issue from AWS, stack-overflow, etc. All my environments are healthy and functional. I have migrated to nf Tower thinking this would bypass or resolve my issues but they still persist. I don't know if this is the correct place to post this but I am desperate for help now. Thank you in advance!
Hello @kkerns85 , in my experience troubleshooting this kind of problem is "part art, part science" given the number of inter-connecting factors. We'd be happy to assist if you open an issue at https://github.com/seqeralabs/nf-tower/issues and can provide more details on your setup.
One initial suggestion I have for you is to check the underlying ECS clusters once your Jobs become Runnable - are Worker instances able to join the cluster(s), or does the membership count remain at 0?
One initial suggestion I have for you is to check the underlying ECS clusters once your Jobs become Runnable - are Worker instances able to join the cluster(s), or does the membership count remain at 0?
4 replies
Regarding nf-core pipelines, see the docs here: https://nf-co.re/developers/adding_pipelines (basically, join slack and tell us about it in the
#new-pipelines channel)
1 reply
Hi all - we're launching NF Tower actions programmatically. Our workflow runs often share identical parameters for the initial processes, but differ in the parameters used at later ones. Is there a way to always use the
-resume option in NF Tower, so that the outputs from these initial processes can be reused? Thanks!
21 replies
Hi all, I'm attempting to build HISAT2 indices for a large genome as part of the nf-core rnaseq pipeline. This is a high-memory, long running type of job since it's a big genome. It's a non-reference organisms, so I can't download these indices. I used tower to set up my compute environment on AWS batch with mainly default parameters. This particular job is getting stuck in the "submitted" state, and based on conversation with the slack channel, we're thinking this may be a scheduler / resource issue. Has anyone else run into something similar and know how to address it?
2 replies
I cant see launchpad and workspace , if I deployed it on local system.. http://localhost8000 .How to setup this on local?
The open source version is not the same as the version that runs at https://tower.nf - it's possible that it doesn't have these features
Which are the api's supported here - http://localhost:8080 ? I can see only GET api's like http://localhost:8080/service-info?token, http://localhost:8080/workflow/list?token
Has anyone come across (or documented) a breakdown of what capabilities are available between hosted, community and enterprise deployments of nextflow tower? Thank you! https://help.tower.nf/getting-started/deployments/
Off the top of my head I think that the hosted / enterprise editions can launch pipelines, have organisational structures with multi-user access and roles, workspaces for organising runs and credentials and so on. The community edition is basically just for monitoring runs that you launch yourself, and you can't really share those runs with anyone.
3 replies
(this is pretty much what those docs say too, just a little more verbose)
And a bunch of other stuff that I haven't mentioned probably.
Hi all, I'm testing Tower to see the capabilities of the tool and it's truly awesome. One doubt that I've is, is it possible to describe the same detail level for the nextflow CLI the compute environments? I mean, using Tower, I can define some parameters for example to create an Slurm cluster: login hostname, login port, etc. How these parameters could be used on a CLI Nextflow pipeline? Using executors? Thanks in advance!
for example, nextlow does not need to be aware of slurm login host, therefore there isn't such option
thanks for the answer @pditommaso , and how can we approach something similar without using Tower? for example, having a central node with Nextflow to deploy processes or pipelines against remote Slurm clusters? Maybe a wrapper or a custom script?
21 replies
Many many thanks for your answers again @ewels , and of course, congratule always the great work of the devs on the Nextflow/Tower tools because are truly awesome without doubts
Hi there, thanks for the great tool.
I’ve noticed lately that the jobs remain in the blue running state, even though they have been cancelled and are not running anymore. Failed jobs and successfully completed jobs are accurately reflected, however. For reference, I am running all my jobs on SLURM. I was wondering if there is maybe something I am doing wrong on my part or if there is any parameter I should change to fix this? I noticed this happening about a month ago. Thanks!
including the workflow id, and the nextflow logs
@pditommaso an "open in new tab" button would be nice :eyes: :sweat_smile:
I’ve been looking into Nextflow and Tower and managed to install and run Tower on a cloud VM using a cloud SQL database. Seems very powerful! Monitoring workflows works fine but I can’t seem to get to the API. The service-info endpoint works:
$ curl -H "Authorization: Bearer TOKEN” http://0.0.0.0:8000/api/service-info
{"message":null,"serviceInfo":{"version":"20.06.0","commitId":null,"authTypes":[],"loginPath":"/login"}}But any other endpoint gives 401 unauthorized:
$ curl -H "Authorization: Bearer TOKEN” http://0.0.0.0:8000/api/workflow
frontend_1 | 172.18.0.1 - - [07/Feb/2022:18:27:18 +0000] "GET /api/workflow HTTP/1.1" 401 5 "-" "curl/7.68.0"I’m using the tokens I get from the Your tokens page in Tower. Am I missing something in the API call?
Useful tool: http://isup.me/tower.nf
Can anyone suggest any way to debug why on runs 'Resolved configuration' results in n/a on tower.nf?
Myself and a colleague have the same settings (nxf version, same workspace organisation etc), by my configuration always resolves to n/a but theirs lists all the parameters
(screenshot coming)
That's actually why I asking if there are any tips to debug, because I can't find a way to replicate it
No-one else seems to be having the problem (as far as I found so far). But it also happens to me when running locally from my laptop as well, and sent to my prviate workspace (not just the workspace/organisation)
Hi all, I'm new to Nextflow Tower CLI. I'm testing it and it's very cool! I'm wondering how to get detailed info af a run. I've tried:
tw runs view --id=1R5EyNgnXGTxXGbut I get ERROR: Missing the required parameter 'launchId' when calling describeLaunch. What I'm doing wrong? Thanks!