Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 09:42

    junyk on nextflow-basic

    Turn off ansi log Allow Nextflow executable path … (compare)

  • 02:00

    junyk on master

    Add unit tests to tools involve… Merge branch 'master' of https:… Merge branch 'master' of https:… and 4 more (compare)

  • 02:00
    junyk closed #100
  • Jun 15 05:59

    junyk on master

    upgrade to release v0.11.7, inc… Merge branch 'master' into rele… Merge pull request #58 from PMC… (compare)

  • Jun 15 05:59
    junyk closed #58
  • Jun 15 05:59
    junyk synchronize #58
  • Jun 15 05:59

    junyk on release-v0.11.7

    Apply janis pre-hook command update script path Execute pre janis hook from com… and 17 more (compare)

  • Jun 15 05:58
    junyk opened #58
  • Jun 15 05:57

    junyk on master

    run tests with pytest update version number add pytest to test requirements and 2 more (compare)

  • Jun 15 05:57
    junyk closed #79
  • Jun 15 05:57
    junyk synchronize #79
  • Jun 15 05:57

    junyk on release-v0.11.4

    Add fromwdl ingestor Add ForEachSelector Add _foreach to workflow (unimp… and 23 more (compare)

  • Jun 15 05:57
    junyk opened #79
  • Jun 15 05:52

    junyk on nextflow-basic

    Fix python tool output logic Add unit tests for python tool (compare)

  • Jun 15 01:16

    junyk on nextflow-basic

    Allow nextflow caching (compare)

  • Jun 11 08:16
    jiaan-yu opened #107
  • Jun 11 07:47
    jiaan-yu synchronize #25
  • Jun 11 07:21
    jiaan-yu synchronize #25
  • Jun 11 07:14
    jiaan-yu synchronize #25
  • Jun 11 07:07
    jiaan-yu synchronize #25
KerstenBreuer
@KerstenBreuer
Could you maybe test it when you have time and give me some feedback?
KerstenBreuer
@KerstenBreuer
Please note that there are currently following limitations:
  • the translation into both WDL and CWL works, however, the execution of WDL workflows is not yet functional
  • CWLab only supports the standard CWLTypes and their WDL equivalents. Also arrays of these types are supported. However, nested arrays (arrays of arrays of CWLTypes) do not work. I need come up with a nice visualization in the HTML input form for this specific case.
Michael Franklin
@illusional

That’s awesome @KerstenBreuer! The nested arrays is a tough one haha! We’re having the same trouble with passing inputs in via the CLI.

I should have some time this week, so I’ll let you know how I go!

KerstenBreuer
@KerstenBreuer
Thanks a lot @illusional
By the way, the code you originally commited was move to (https://github.com/CompEpigen/CWLab/blob/janis_wdl/cwlab/wf_input/read_janis.py#L14-L51).
I tested it with the latest release of Janis and it seems to work perfectly. Could you double check? :)
Of note, I used not all the functions you commited as I don't need to directly reads input parameters from the janis script anymore.
Bernie Pope
@bjpop
How do I tell Janis to run a pipeline from the registry? I can't seem to find any docs about that
Evan Thomas
@evan-wehi
How do I specify walltime for a job step? There seems to be a default of 23 hours in cromwellconfiguration.py but I need to be able to set in the pipeline definition.
Michael Franklin
@illusional
Currently you can't, it's set to 24 hours as historically it has played best with cloud and other HPC systems. It's something we'd like a CommandTool to be able to specify, and a user to override (like CPU + memory resources).
Ideally, it would be good to bring in the ability to write expressions (from the conditionals branch) + add a mechanism for specifying storage requirements, these are often derived from inputs.
Evan Thomas
@evan-wehi
I’ve got steps timing out at 23 hours and others that only take a few minutes. Our HPC is busy so I’m also competing for resources ATM so it’s important for me to specify walltimes. I’ll just change cromwellconfiguration.py in the short term.
Michael Franklin
@illusional
Yeah, I see the value in variable expected walltimes for tasks, makes scheduling tasks a lot more efficient. Although I think you should be able to specify job lengths over 1 day, the general advice we'd give (same with Pawsey) is to break the task up if possible by regions, or scale the resources. Not always possible though I understand.
Evan Thomas
@evan-wehi

Occasionally, maybe 1% of the time workflows hang with the janis process showing 100% CPU. I typically have hundreds of workflows in flight at a time so I sometimes need to query them to see what state they are in. janis watch to the rescue! Unfortunately, I go an exception and I’m not sure how to interpret the output.

WID:        c46846
EngId:      26ae5e95-93aa-43fa-93a5-620af67b935c
Name:       tcga_exome_caller
Engine:     cromwell (localhost:33076) [PID=9325]

Task Dir:   /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/
Exec Dir:   /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-5665-01A-01D-1650-10--TCGA-A6-5665-10A-01D-1650-10/janis/execution/tcga_exome_caller/26ae5e95-93aa-43fa-93a5-620af67b935c

Status:     Failed
Duration:   22h:40m:06s
Start:      2020-02-16T23:37:39.049823+00:00
Finish:     N/A
Updated:    6s ago (2020-02-17T22:17:38+00:00)

Jobs: 
    [✓] tumour_dedup_step (41m:49s)
    [✓] normal_dedup_step (1h:09m:16s)
    [✓] tumour_dedup_index (4h:12m:58s)
    [✓] normal_dedup_index (9h:29m:16s)
    [~] sequenza_bam2seqz (12h:00m:31s)
            batchid: 16699000
    [✓] run_snp_pileup (1h:22m:41s)       


Error: Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 397, in resume
    cb = self.main_queue.get(False)
  File "/stornext/System/data/apps/python/python-3.7.0/lib/python3.7/queue.py", line 167, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 404, in resume
    if self.database.workflowmetadata.please_abort:
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 25, in __getattr__
    return self[item]
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/keyvaluedbproviderbase.py", line 35, in __getitem__
    return self.kvdb.__getitem__(item)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 243, in __getitem__
    item = self.conn.select_one(GET_ITEM, (key,))
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 515, in select_one
    return next(iter(self.select(req, arg)))
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 507, in select
    self.check_raise_error()
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 475, in check_raise_error
    reraise(e_type, e_value, e_tb)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 71, in reraise
    raise value
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/sqlitedict.py", line 409, in run
    cursor.execute(req, arg)
sqlite3.OperationalError: unable to open database file

It says Status: Failed but there is a job still running.

Evan Thomas
@evan-wehi
As an update - that was my very first attempt with janis watch. I’ve been querying a number of other runs and it has been working very well.
Michael Franklin
@illusional

Glad to hear it's working well! This looks like one of the SQLite / concurrently reading the databae issues that we've been seeing. Do you see anything strange in the engine log?

$outputdir/janis/logs/engine.log

Evan Thomas
@evan-wehi
Nothing obvious. I don’t think there should be a race because only one cromwell instance should be accessing it? Unless I somehow launched the same workflow twice.
I can zip up the janis directory and give it to you if that helps. As I mentioned, it turns out to be a uncommon error.
Michael Franklin
@illusional
It may happen when the "watch" reads the same database and throws an error. I've seen it here a bit, and have some strategies to address it soon.
Evan Thomas
@evan-wehi

Here is a feature I just realised I would really like to have - I don’t think’s hard and janis watch may already do it. In my use, case I’m running 10^2-10^3 workflows at once with very variable runtimes. Inevitably, some of them fail (for all sorts of reasons). My practice has been to led a running cohort finish and the resubmit from the top. This means I might be waiting 2 days for one or two runs to finish before I can resubmit to pick up the failed runs. My logic can only tell if a workflow has finished.

What would be really handy is if I can detect all the states of the workflow, i.e. NOT STARTED/DOESN’T EXIST, RUNNING, FAILED, COMPLETE. This is information is essentially in janis watch but I need an API that I can call from bash script.

Michael Franklin
@illusional
Yeah I like that idea!
Evan Thomas
@evan-wehi
janis watch —status?
Michael Franklin
@illusional

We have a fairly close analogue at the moment, just less specific. You can call

janis metadata <wid / folder>

And it returns a table with known metadata about the workflow (including status and last_updated)

Evan Thomas
@evan-wehi
That’s probably all I need.
Evan Thomas
@evan-wehi
I think so. Presumably the status line will alway be there (if the workflow exists). What values can it take?
Michael Franklin
@illusional

It's just using tabulate to generate the structure, so it should be fairly consistent. We could potentially add a "format-option" or something to allow you to provide a different tabulate format option.

Generating code: https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/management/workflowmanager.py#L966

Keys: https://github.com/PMCC-BioinformaticsCore/janis-assistant/blob/master/janis_assistant/data/enums/workflowmetadatakeys.py

Not every key is guaranteed to be there though.
Michael Franklin
@illusional
Yep!
Evan Thomas
@evan-wehi
$ janis metadata <dir>|grep status|tr -s ' '|cut -d ' ' -f 2
Failed
Michael Franklin
@illusional
You'll probably get the __str__ state of it, so
    def __str__(self):
        __str = {
            TaskStatus.PROCESSING.value: "Processing",
            TaskStatus.QUEUED.value: "Queued",
            TaskStatus.RUNNING.value: "Running",
            TaskStatus.COMPLETED.value: "Completed",
            TaskStatus.FAILED.value: "Failed",
            TaskStatus.ABORTED.value: "Terminated",
            TaskStatus.DRY_RUN.value: "Dry run",
            TaskStatus.ON_HOLD.value: "On hold",
        }
        return __str[self.value]
Evan Thomas
@evan-wehi
Got it.
Evan Thomas
@evan-wehi
What’s the difference between “Processing” and “Running”? What does “Terminated” mean? Killed by the operator ?
Michael Franklin
@illusional

Workflows automatically get the queued when they're submitted. Processing means it's starting everything up (like Cromwell + mysql) and submitting to the engine. Running is when it gets a notification from Cromwell saying it's actually running.

Yep, Terminated is the status when you janis abort a workflow.

Evan Thomas
@evan-wehi
Excellent - for my use case I want to detect “Failed” and “Terminated”.
Michael Franklin
@illusional
If Janis is killed suddenly, it might not have enough time to move the workflow into a failed or terminated status. It's worth checking the last_updated_time, if a workflow hasn't been updated in a couple of hours, it's likely Janis has been killed.
Evan Thomas
@evan-wehi
Ok, that’s good to know. I’m mainly concerned with not clobbering a potentially viable workflow. When nothing is running I can scan for workflows in unexpected states.
Evan Thomas
@evan-wehi
I’m getting a hung process in janis metadata. Here is the stack trace when I ^C out of it:
$ /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/python /stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis metadata /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
^C^C2020-02-21T10:20:38 [CRITICAL]: Error when opening DB connection to: /stornext/HPCScratch/PapenfussLab/projects/tcga-data/COAD/TCGA-A6-2684-01A-01D-A270-10--TCGA-A6-2684-10A-01D-A270-10
Traceback (most recent call last):
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/bin/janis", line 8, in <module>
    sys.exit(process_args())
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 130, in process_args
    return cmds[args.command](args)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/cli.py", line 527, in do_metadata
    tm = ConfigManager.manager().from_wid(wid)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/configmanager.py", line 196, in from_wid
    return WorkflowManager.from_path_get_latest(wid)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowmanager.py", line 253, in from_path_get_latest
    wid = WorkflowDbManager.get_latest_workflow(path=path)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/management/workflowdbmanager.py", line 95, in get_latest_workflow
    runDb = RunDbProvider(db=connection, cursor=connection.cursor())
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/providers/rundbprovider.py", line 21, in __init__
    super().__init__(db, cursor)
  File "/stornext/HPCScratch/PapenfussLab/projects/tcga-exome/venv/lib/python3.7/site-packages/janis_assistant/data/dbproviderbase.py", line 11, in __init__
    self.cursor.execute(schema)
KeyboardInterrupt
The workflow seems to have finished though.
Michael Franklin
@illusional
Hmm that's strange. In the latest version of janis-assistant (currently unreleased), I've made changes to this process for sql to be "readonly" which might make a difference. Weird that it's stuck on executing the query
Evan Thomas
@evan-wehi
I can give you the janis directory. I have at least 2 in this condition.
Evan Thomas
@evan-wehi
My bad - it’s our filesystem.
Evan Thomas
@evan-wehi

I’m getting a lot of sqlite3 exceptions using janis watch and janis metadata. Often, I’ll query a run and it will return OK but on subsequent queries it returns exceptions - the database seems to be borked. I’m losing a lot of runs to this issue and it makes very difficult to know what state my workload is in. I think I’m a release behind.

$ janis --version
--------------------  ------
janis-core            v0.9.4
janis-assistant       v0.9.5
janis-templates       v0.9.2
janis-bioinformatics  v0.9.3
janis-unix            v0.9.0
janis-pipelines       v0.9.1
--------------------  ———

I’m reluctant to upgrade because I have, literally hundreds of runs in flight. But, I think you mentioned, that the newer version is more rubust wrt sqlite. If so, I’m willing to upgrade to the new version.

Michael Franklin
@illusional

Hey @evan-wehi. The latest version of janis-assistant contains fixes for sqlite, specifically when watching a workflow (opening dB in readonly mode). It does also require the latest (and backwards compatible) janis-core and bioinformatics.

I’m not sure the best way to give you these changes if you’re still running pipelines as I don’t really know how python goes like that.

Sticking within the 0.9.x should all be backwards compatible, just sometimes the new version might require fixes to new version of dependencies.

We’re looking to do a release next week with a bunch of bug fixes, so could be worth waiting for that. I’m a little distant for support this week but can be more hands on next

Michael Franklin
@illusional
Hello to all those from the WEHI talk, this chat is for pretty general support. Happy to take any questions about anything pipelines related!
Rebecca Louise Evans
@beccyl
I'm trying to create a Janis tool equivalent to a CWL CommandLineTool I already have.
I have a script which takes a file as input, and generates a numeric value as output (to stdout). eg. running tool generates "21" (to stdout) and the int 21 is the input to another tool (not as a File input but as a parameter int - eg '-s 21'. In CWL this is done by loadContents: true, and outputEval: '$(parseInt(self[0].contents,10))'. Without rewriting the tool (which is a shell script using bc) is there anyway to direct ToolOutput to Int with Selector stdout. ??
Juny Kesumadewi
@junyk
Hi @beccyl , that's a good question. In Janis, you can use the .contents().as_int() to convert a stdout content to an int. I have added a dummy workflow example on janis-patterns https://github.com/PMCC-BioinformaticsCore/janis-patterns/blob/master/outputs/stdout_as_int.py#L47. Hope that helps!
Rebecca Louise Evans
@beccyl
Thanks @junyk