janis-pipelines/community

Michael Franklin

@illusional

Good afternoon! Version 0.9.0 of Janis (+ related) were all released on Friday. I'd encourage checking out the release notes as there are a few backwards incompatible changes. Please drop a line here if you're having any trouble.

https://github.com/PMCC-BioinformaticsCore/janis/releases/tag/v0.9.1

KerstenBreuer

@KerstenBreuer

Hi Michael,
thanks for your email. I had time to make some progress on the integration of Janis into CWLab.
It's currently on branch https://github.com/CompEpigen/CWLab/tree/janis_wdl. The image above shows a screenshot of the import GUI.

Could you maybe test it when you have time and give me some feedback?

KerstenBreuer

@KerstenBreuer

Please note that there are currently following limitations:

the translation into both WDL and CWL works, however, the execution of WDL workflows is not yet functional
CWLab only supports the standard CWLTypes and their WDL equivalents. Also arrays of these types are supported. However, nested arrays (arrays of arrays of CWLTypes) do not work. I need come up with a nice visualization in the HTML input form for this specific case.

Michael Franklin

@illusional

That’s awesome @KerstenBreuer! The nested arrays is a tough one haha! We’re having the same trouble with passing inputs in via the CLI.

I should have some time this week, so I’ll let you know how I go!

KerstenBreuer

@KerstenBreuer

Thanks a lot @illusional

By the way, the code you originally commited was move to (https://github.com/CompEpigen/CWLab/blob/janis_wdl/cwlab/wf_input/read_janis.py#L14-L51).

I tested it with the latest release of Janis and it seems to work perfectly. Could you double check? :)

Of note, I used not all the functions you commited as I don't need to directly reads input parameters from the janis script anymore.

Bernie Pope

@bjpop

How do I tell Janis to run a pipeline from the registry? I can't seem to find any docs about that

I want to run the https://janis.readthedocs.io/en/latest/pipelines/wgssomaticmulticallers.html pipeline

Evan Thomas

@evan-wehi

How do I specify walltime for a job step? There seems to be a default of 23 hours in cromwellconfiguration.py but I need to be able to set in the pipeline definition.

Michael Franklin

@illusional

Currently you can't, it's set to 24 hours as historically it has played best with cloud and other HPC systems. It's something we'd like a CommandTool to be able to specify, and a user to override (like CPU + memory resources).

Ideally, it would be good to bring in the ability to write expressions (from the conditionals branch) + add a mechanism for specifying storage requirements, these are often derived from inputs.

Evan Thomas

@evan-wehi

I’ve got steps timing out at 23 hours and others that only take a few minutes. Our HPC is busy so I’m also competing for resources ATM so it’s important for me to specify walltimes. I’ll just change cromwellconfiguration.py in the short term.

Michael Franklin

@illusional

Yeah, I see the value in variable expected walltimes for tasks, makes scheduling tasks a lot more efficient. Although I think you should be able to specify job lengths over 1 day, the general advice we'd give (same with Pawsey) is to break the task up if possible by regions, or scale the resources. Not always possible though I understand.

Evan Thomas

@evan-wehi

Occasionally, maybe 1% of the time workflows hang with the janis process showing 100% CPU. I typically have hundreds of workflows in flight at a time so I sometimes need to query them to see what state they are in. janis watch to the rescue! Unfortunately, I go an exception and I’m not sure how to interpret the output.

WID:        c46846
EngId:      26ae5e95-93aa-43fa-93a5-620af67b935c
Name:       tcga_exome_caller
Engine:     cromwell (localhost:33076) [PID=9325]

Task Dir:   /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/
Exec Dir:   /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/janis/execution/tcga_exome_caller/26ae5e95-93aa-43fa-93a5-620af67b935c

Status:     Failed
Duration:   22h:40m:06s
Start:      2020-02-16T23:37:39.049823+00:00
Finish:     N/A
Updated:    6s ago (2020-02-17T22:17:38+00:00)

Jobs: 
    [✓] tumour_dedup_step (41m:49s)
    [✓] normal_dedup_step (1h:09m:16s)
    [✓] tumour_dedup_index (4h:12m:58s)
    [✓] normal_dedup_index (9h:29m:16s)
    [~] sequenza_bam2seqz (12h:00m:31s)
            batchid: 16699000
    [✓] run_snp_pileup (1h:22m:41s)       


Error: Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 397, in resume
    cb = self.main_queue.get(False)
  File "/stornext/System/data/apps/python/python-3.7.0/lib/python3.7/queue.py", line 167, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 404, in resume
    if self.database.workflowmetadata.please_abort:
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 25, in __getattr__
    return self[item]
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 35, in __getitem__
    return self.kvdb.__getitem__(item)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 243, in __getitem__
    item = self.conn.select_one(GET_ITEM, (key,))
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 515, in select_one
    return next(iter(self.select(req, arg)))
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 507, in select
    self.check_raise_error()
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 475, in check_raise_error
    reraise(e_type, e_value, e_tb)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 71, in reraise
    raise value
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 409, in run
    cursor.execute(req, arg)
sqlite3.OperationalError: unable to open database file

It says Status: Failed but there is a job still running.

Evan Thomas

@evan-wehi

As an update - that was my very first attempt with janis watch. I’ve been querying a number of other runs and it has been working very well.

Michael Franklin

@illusional

Glad to hear it's working well! This looks like one of the SQLite / concurrently reading the databae issues that we've been seeing. Do you see anything strange in the engine log?

$outputdir/janis/logs/engine.log

Evan Thomas

@evan-wehi

Nothing obvious. I don’t think there should be a race because only one cromwell instance should be accessing it? Unless I somehow launched the same workflow twice.

I can zip up the janis directory and give it to you if that helps. As I mentioned, it turns out to be a uncommon error.

Michael Franklin

@illusional

It may happen when the "watch" reads the same database and throws an error. I've seen it here a bit, and have some strategies to address it soon.

Evan Thomas

@evan-wehi

Here is a feature I just realised I would really like to have - I don’t think’s hard and janis watch may already do it. In my use, case I’m running 10^2-10^3 workflows at once with very variable runtimes. Inevitably, some of them fail (for all sorts of reasons). My practice has been to led a running cohort finish and the resubmit from the top. This means I might be waiting 2 days for one or two runs to finish before I can resubmit to pick up the failed runs. My logic can only tell if a workflow has finished.

What would be really handy is if I can detect all the states of the workflow, i.e. NOT STARTED/DOESN’T EXIST, RUNNING, FAILED, COMPLETE. This is information is essentially in janis watch but I need an API that I can call from bash script.

Michael Franklin

@illusional

Yeah I like that idea!

Evan Thomas

@evan-wehi

janis watch —status?

Michael Franklin

@illusional

We have a fairly close analogue at the moment, just less specific. You can call

janis metadata <wid / folder>

And it returns a table with known metadata about the workflow (including status and last_updated)

Evan Thomas

@evan-wehi

That’s probably all I need.

Evan Thomas

@evan-wehi

I think so. Presumably the status line will alway be there (if the workflow exists). What values can it take?

Michael Franklin

@illusional

It's just using tabulate to generate the structure, so it should be fairly consistent. We could potentially add a "format-option" or something to allow you to provide a different tabulate format option.

Generating code: https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/management/workflowmanager.py#L966

Keys: https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/data/enums/workflowmetadatakeys.py

Not every key is guaranteed to be there though.

Evan Thomas

@evan-wehi

And these are the possible states?
https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/data/enums/taskstatus.py

Michael Franklin

@illusional

Yep!

Evan Thomas

@evan-wehi

$ janis metadata <dir>|grep status|tr -s ' '|cut -d ' ' -f 2
Failed

Michael Franklin

@illusional

You'll probably get the __str__ state of it, so

    def __str__(self):
        __str = {
            TaskStatus.PROCESSING.value: "Processing",
            TaskStatus.QUEUED.value: "Queued",
            TaskStatus.RUNNING.value: "Running",
            TaskStatus.COMPLETED.value: "Completed",
            TaskStatus.FAILED.value: "Failed",
            TaskStatus.ABORTED.value: "Terminated",
            TaskStatus.DRY_RUN.value: "Dry run",
            TaskStatus.ON_HOLD.value: "On hold",
        }
        return __str[self.value]

Evan Thomas

@evan-wehi

Got it.

Evan Thomas

@evan-wehi

What’s the difference between “Processing” and “Running”? What does “Terminated” mean? Killed by the operator ?

Michael Franklin

@illusional

Workflows automatically get the queued when they're submitted. Processing means it's starting everything up (like Cromwell + mysql) and submitting to the engine. Running is when it gets a notification from Cromwell saying it's actually running.

Yep, Terminated is the status when you janis abort a workflow.

Evan Thomas

@evan-wehi

Excellent - for my use case I want to detect “Failed” and “Terminated”.

Michael Franklin

@illusional

If Janis is killed suddenly, it might not have enough time to move the workflow into a failed or terminated status. It's worth checking the last_updated_time, if a workflow hasn't been updated in a couple of hours, it's likely Janis has been killed.

Evan Thomas

@evan-wehi

Ok, that’s good to know. I’m mainly concerned with not clobbering a potentially viable workflow. When nothing is running I can scan for workflows in unexpected states.

Evan Thomas

@evan-wehi

I’m getting a hung process in janis metadata. Here is the stack trace when I ^C out of it:

$ /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/python /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis metadata /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
^C^C2020-02-21T10:20:38 [CRITICAL]: Error when opening DB connection to: /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis", line 8, in <module>
    sys.exit(process_args())
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 130, in process_args
    return cmds[args.command](args)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 527, in do_metadata
    tm = ConfigManager.manager().from_wid(wid)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/configmanager.py", line 196, in from_wid
    return WorkflowManager.from_path_get_latest(wid)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 253, in from_path_get_latest
    wid = WorkflowDbManager.get_latest_workflow(path=path)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowdbmanager.py", line 95, in get_latest_workflow
    runDb = RunDbProvider(db=connection, cursor=connection.cursor())
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/providers/rundbprovider.py", line 21, in __init__
    super().__init__(db, cursor)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/dbproviderbase.py", line 11, in __init__
    self.cursor.execute(schema)
KeyboardInterrupt

The workflow seems to have finished though.

Michael Franklin

@illusional

Hmm that's strange. In the latest version of janis-assistant (currently unreleased), I've made changes to this process for sql to be "readonly" which might make a difference. Weird that it's stuck on executing the query

Evan Thomas

@evan-wehi

I can give you the janis directory. I have at least 2 in this condition.

Evan Thomas

@evan-wehi

My bad - it’s our filesystem.

Evan Thomas

@evan-wehi

I’m getting a lot of sqlite3 exceptions using janis watch and janis metadata. Often, I’ll query a run and it will return OK but on subsequent queries it returns exceptions - the database seems to be borked. I’m losing a lot of runs to this issue and it makes very difficult to know what state my workload is in. I think I’m a release behind.

$ janis --version
--------------------  ------
janis-core            v0.9.4
janis-assistant       v0.9.5
janis-templates       v0.9.2
janis-bioinformatics  v0.9.3
janis-unix            v0.9.0
janis-pipelines       v0.9.1
--------------------  ———

I’m reluctant to upgrade because I have, literally hundreds of runs in flight. But, I think you mentioned, that the newer version is more rubust wrt sqlite. If so, I’m willing to upgrade to the new version.

Michael Franklin

@illusional

Hey @evan-wehi. The latest version of janis-assistant contains fixes for sqlite, specifically when watching a workflow (opening dB in readonly mode). It does also require the latest (and backwards compatible) janis-core and bioinformatics.

I’m not sure the best way to give you these changes if you’re still running pipelines as I don’t really know how python goes like that.

Sticking within the 0.9.x should all be backwards compatible, just sometimes the new version might require fixes to new version of dependencies.

We’re looking to do a release next week with a bunch of bug fixes, so could be worth waiting for that. I’m a little distant for support this week but can be more hands on next

Michael Franklin

@illusional

Hello to all those from the WEHI talk, this chat is for pretty general support. Happy to take any questions about anything pipelines related!

Where communities thrive

People

Repo info

Activity