rlupat on master
Update __meta__.py Update jani… (compare)
rlupat on master
Update __meta__.py Update jani… (compare)
rlupat on master
Seperate add_inputs_facets from… Fix unit test Merge pull request #38 from jia… (compare)
rlupat on master
Add tool LocaliseFolder Merge branch 'master' of https:… Merge pull request #15 from jia… (compare)
rlupat on master
Add workflow RNASeqGeneExpressi… Remove redundant options and fi… Add more arguments to htseq-cou… and 17 more (compare)
rlupat on master
Reformat file with black Add tool HTSeqCount Merge Star tool and 11 more (compare)
Occasionally, maybe 1% of the time workflows hang with the janis process showing 100% CPU. I typically have hundreds of workflows in flight at a time so I sometimes need to query them to see what state they are in. janis watch
to the rescue! Unfortunately, I go an exception and I’m not sure how to interpret the output.
WID: c46846
EngId: 26ae5e95-93aa-43fa-93a5-620af67b935c
Name: tcga_exome_caller
Engine: cromwell (localhost:33076) [PID=9325]
Task Dir: /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/
Exec Dir: /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/janis/execution/tcga_exome_caller/26ae5e95-93aa-43fa-93a5-620af67b935c
Status: Failed
Duration: 22h:40m:06s
Start: 2020-02-16T23:37:39.049823+00:00
Finish: N/A
Updated: 6s ago (2020-02-17T22:17:38+00:00)
Jobs:
[✓] tumour_dedup_step (41m:49s)
[✓] normal_dedup_step (1h:09m:16s)
[✓] tumour_dedup_index (4h:12m:58s)
[✓] normal_dedup_index (9h:29m:16s)
[~] sequenza_bam2seqz (12h:00m:31s)
batchid: 16699000
[✓] run_snp_pileup (1h:22m:41s)
Error: Traceback (most recent call last):
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 397, in resume
cb = self.main_queue.get(False)
File "/stornext/System/data/apps/python/python-3.7.0/lib/python3.7/queue.py", line 167, in get
raise Empty
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 404, in resume
if self.database.workflowmetadata.please_abort:
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 25, in __getattr__
return self[item]
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 35, in __getitem__
return self.kvdb.__getitem__(item)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 243, in __getitem__
item = self.conn.select_one(GET_ITEM, (key,))
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 515, in select_one
return next(iter(self.select(req, arg)))
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 507, in select
self.check_raise_error()
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 475, in check_raise_error
reraise(e_type, e_value, e_tb)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 71, in reraise
raise value
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 409, in run
cursor.execute(req, arg)
sqlite3.OperationalError: unable to open database file
It says Status: Failed
but there is a job still running.
Here is a feature I just realised I would really like to have - I don’t think’s hard and janis watch
may already do it. In my use, case I’m running 10^2-10^3 workflows at once with very variable runtimes. Inevitably, some of them fail (for all sorts of reasons). My practice has been to led a running cohort finish and the resubmit from the top. This means I might be waiting 2 days for one or two runs to finish before I can resubmit to pick up the failed runs. My logic can only tell if a workflow has finished.
What would be really handy is if I can detect all the states of the workflow, i.e. NOT STARTED/DOESN’T EXIST, RUNNING, FAILED, COMPLETE. This is information is essentially in janis watch
but I need an API that I can call from bash script.
It's just using tabulate to generate the structure, so it should be fairly consistent. We could potentially add a "format-option" or something to allow you to provide a different tabulate format option.
Generating code: https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/management/workflowmanager.py#L966
__str__
state of it, so def __str__(self):
__str = {
TaskStatus.PROCESSING.value: "Processing",
TaskStatus.QUEUED.value: "Queued",
TaskStatus.RUNNING.value: "Running",
TaskStatus.COMPLETED.value: "Completed",
TaskStatus.FAILED.value: "Failed",
TaskStatus.ABORTED.value: "Terminated",
TaskStatus.DRY_RUN.value: "Dry run",
TaskStatus.ON_HOLD.value: "On hold",
}
return __str[self.value]
Workflows automatically get the queued when they're submitted. Processing means it's starting everything up (like Cromwell + mysql) and submitting to the engine. Running is when it gets a notification from Cromwell saying it's actually running.
Yep, Terminated is the status when you janis abort
a workflow.
janis metadata
. Here is the stack trace when I ^C out of it:$ /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/python /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis metadata /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
^C^C2020-02-21T10:20:38 [CRITICAL]: Error when opening DB connection to: /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
Traceback (most recent call last):
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis", line 8, in <module>
sys.exit(process_args())
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 130, in process_args
return cmds[args.command](args)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 527, in do_metadata
tm = ConfigManager.manager().from_wid(wid)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/configmanager.py", line 196, in from_wid
return WorkflowManager.from_path_get_latest(wid)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 253, in from_path_get_latest
wid = WorkflowDbManager.get_latest_workflow(path=path)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowdbmanager.py", line 95, in get_latest_workflow
runDb = RunDbProvider(db=connection, cursor=connection.cursor())
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/providers/rundbprovider.py", line 21, in __init__
super().__init__(db, cursor)
File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/dbproviderbase.py", line 11, in __init__
self.cursor.execute(schema)
KeyboardInterrupt
I’m getting a lot of sqlite3 exceptions using janis watch
and janis metadata
. Often, I’ll query a run and it will return OK but on subsequent queries it returns exceptions - the database seems to be borked. I’m losing a lot of runs to this issue and it makes very difficult to know what state my workload is in. I think I’m a release behind.
$ janis --version
-------------------- ------
janis-core v0.9.4
janis-assistant v0.9.5
janis-templates v0.9.2
janis-bioinformatics v0.9.3
janis-unix v0.9.0
janis-pipelines v0.9.1
-------------------- ———
I’m reluctant to upgrade because I have, literally hundreds of runs in flight. But, I think you mentioned, that the newer version is more rubust wrt sqlite. If so, I’m willing to upgrade to the new version.
Hey @evan-wehi. The latest version of janis-assistant contains fixes for sqlite, specifically when watching a workflow (opening dB in readonly mode). It does also require the latest (and backwards compatible) janis-core and bioinformatics.
I’m not sure the best way to give you these changes if you’re still running pipelines as I don’t really know how python goes like that.
Sticking within the 0.9.x should all be backwards compatible, just sometimes the new version might require fixes to new version of dependencies.
We’re looking to do a release next week with a bunch of bug fixes, so could be worth waiting for that. I’m a little distant for support this week but can be more hands on next
.contents().as_int()
to convert a stdout content to an int. I have added a dummy workflow example on janis-patterns https://github.com/PMCC-BioinformaticsCore/janis-patterns/blob/master/outputs/stdout_as_int.py#L47. Hope that helps!
I'm trying to automatically extract the sample name from fastq input, then use that sample name to name the output file from fastqc. I've written a GetSampleNameFromFastq pythontool to extract this info, but then it looks like I can't use the output to name the file, I'm getting:
Exception: Unrecognised python file when getting workflow / command tool: tools/celseq-pipeline.py :: Invalid type for output_name: StepOutputSelector
Here's my output statement:
w.output(
'out',
source=w.fastqc.out,
output_name=w.samplename.out,
output_folder=['fastqc']
)
Is there a way to extract the string value from w.samplename.out
? I've tried StringFormatter
and calling to_string_formatter
. The output type of GetSampleNameFromFastq
is a string:
return [TOutput('out', str)]
Hi @marekcmero_twitter , unfortunately the output naming via that mechanism currently only supports variable that comes from workflow input.
The output structure is currently defined by Janis at the start of the workflow execution, and it will not have access to any outputs generated in the middle of the workflow (as the execution is handed over to cromwell/cwltool). It might be possible to re-architecture this to support deciding output name at the end of execution, but we will have to look into this further.
Alternative ways to make your use-cases work:
Any better ideas @illusional ?