common-workflow-language/cwlviewer

https://view.commonwl.org/

Michael R. Crusoe

@mr-c

Should the steps, inputs & output be included? Or should they be replaced with a link to the "full" view?

Chris Hunter

@only1chunts

Thanks for thinking I might be a developer, but sadly you've just gone way past my knowledge on html (or programming of any sort). If developing a widget viewer is not something anyone else needs/wants and its not on your todo list then thats fine, we'll just have to stick with the URL link for now.

Michael R. Crusoe

@mr-c

@only1chunts You can load the URL into an iframe the same way the other widget works today. Later we'll add an ?embed=true mode that is better formatted

Chris Hunter

@only1chunts

perfect, thanks

Stian Soiland-Reyes

@stain

Would something like

also be cool on the CWL Viewer?
Is it too late to make GSOC2019 suggestions for CWL, @mr-c ?

Michael R. Crusoe

@mr-c

@stain nope, but needs to get up asap to attract student talent

Michael R. Crusoe

@mr-c

@stain Also, that badge looks great!

Michael R. Crusoe

@mr-c

https://view.commonwl.org/workflows/github.com/ddbj/human-reseq/blob/master/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl

@stain time to reboot?

Error: cwltool version failed to run on this workflow:
! Cwltool ran successfully, but an unexpected error occurred in CWLViewer!
Help us by reporting it on Gitter or a Github issue

Stian Soiland-Reyes

@stain

hmm.. I get the same error as with cwltool --validate really.
Viewer:

Tool definition failed validation:
../data/git/aa420273539ad7bcf384c37614f6f636810042f2/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl:170:16: Workflow step output 'vcf_tbi' does not correspond to
../data/git/aa420273539ad7bcf384c37614f6f636810042f2/Tools/gatk3-HaplotypeCaller-XCORE1-ploidy1.cwl:51:1:   tool output (expected 'vcf', 'log')

Command line:

(base) stain@biggie:~$ cwltool --validate https://raw.githubusercontent.com/ddbj/human-reseq/master/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl
/home/stain/miniconda3/bin/cwltool 1.0.20190228200117
Tool definition failed validation:
https://raw.githubusercontent.com/ddbj/human-reseq/master/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl:170:16: Workflow step output 'vcf_tbi' does not correspond to
https://raw.githubusercontent.com/ddbj/human-reseq/master/Tools/gatk3-HaplotypeCaller-XCORE1-ploidy1.cwl:51:1:   tool output (expected 'vcf', 'log')

btw, that badge is just https://img.shields.io/badge/cwl-valid-B5314C.svg?logo=common-workflow-language but the viewer would need to have an in-between resource that checks it live.

Michael R. Crusoe

@mr-c

@stain both shields.io and badget.net support json endpoints

https://shields.io/endpoint & https://badgen.net/docs/https

(shields is thought to have timeout issues; badgen is said to be faster in rendering)

Stian Soiland-Reyes

@stain

I suggested that as well on https://github.com/OBF/GSoC/pull/44/files (feel free to modulate)

Michael R. Crusoe

@mr-c

@stain I think this was the commit making the problem I reported earlier? But now it works! https://github.com/ddbj/human-reseq/blob/9064c301d858a796f8340d1e762e5916fd199da0/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl

Stian Soiland-Reyes

@stain

cwltool --validate https://raw.githubusercontent.com/ddbj/human-reseq/9064c301d858a796f8340d1e762e5916fd199da0/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl

gives similar validation error, but this time about secondaryFiles - so in a way it should not have worked! ;)

Michael R. Crusoe

@mr-c

Ping @stain https://view.commonwl.org/workflows/github.com/DataBiosphere/topmed-workflows/blob/master/aligner/sbg-alignment-cwl/topmed-alignment.cwl :-(

Michael R. Crusoe

@mr-c

A CWL v1.0.x to CWL v1.1 standalone converter common-workflow-language/cwl-upgrader#26

Stian Soiland-Reyes

@stain

might it be because of this?

URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
@prefix ns2: <foaf:> .

also the old sbg: 'https://sevenbridges.com'

Stian Soiland-Reyes

@stain

raised as DataBiosphere/topmed-workflows#97 although there is some other bug as https://view.commonwl.org/workflows/github.com/stain/topmed-workflows/blob/namespaces/aligner/sbg-alignment-cwl/topmed-alignment.cwl# turns out empty - seems to be because id confuses the CWL Viewer :(

https://view.commonwl.org/workflows/github.com/stain/topmed-workflows/blob/no-id/aligner/sbg-alignment-cwl/topmed-alignment.cwl works when I deleted id as well

mr-c @mr-c waves hello

Michael R. Crusoe

@mr-c

University of Manchester is having site-wide network issues today, which may mean the CWL Viewer seems unstable. The server is still up, but it's connectivity may come and go for the next hours. There might be intermittent errors if the server tries to check out from git during network outage.

Michael R. Crusoe

@mr-c

Uh oh, every time I use the viewer, the workflow comes out having 0 steps, 0 inputs, and 0 outputs

https://view.commonwl.org/workflows/github.com/yyoshiaki/VIRTUS/blob/master/workflow/virtus-recount.cwl

SWZ

@swzCuroverse

Hi there - cwlviewer seems to down.

Everytime I have used it in the last few days I get "Error: An internal server error occurred"

Peter Amstutz

@tetron

@stain

SWZ

@swzCuroverse

Also, not sure it if it is possible but maybe we can put some of our upcoming CZI money into helping host it if @stain would like that?

Peter Amstutz

@tetron

SWZ

@swzCuroverse

Don't want to step on any toes or anything...obviously

Michael R. Crusoe

@mr-c

I don't think Stian would mind if someone took over the hosting.

SWZ

@swzCuroverse

SWZ

@swzCuroverse

OK, let's see if we can help then and put it into the plan

mr-c @mr-c waves hello to @stain and @tetron and @swZCuroverse

Stian Soiland-Reyes

@stain

@tetron @swzCuroverse could we do a Doodle for when to do that call about the CWL Viewer hosting?

Michael R. Crusoe

@mr-c

Who has the most constrained schedule should suggest the times, yes?

Stian Soiland-Reyes

@stain

I talked with Michael about it now.. https://github.com/common-workflow-language/cwlviewer/blob/main/load.py can be used to load a JSON dump on a fresh CWL Viewer instance and https://github.com/common-workflow-language/cwlviewer/blob/main/dump.sh can make the JSON dump (which is how we do backups)

I just rebooted the server again.. the /tmp had ran out again and the cronjob had failed.. because there was no disk space for docker-compose to do exec! (why!?)

Michael R. Crusoe

@mr-c

@stain how often is the cronjob? Maybe increase the frequency?

Stian Soiland-Reyes

@stain

It's nightly.. but it can run more often because it checks mtime it should not break things in progress

Michael R. Crusoe

@mr-c

maybe twice a day? What does it delete?

Stian Soiland-Reyes

@stain

docker-compose exec -T spring find /tmp -type f -mtime +1 -delete
docker-compose exec -T spring find /tmp -type d -mtime +30 -delete

it's basically deleting the RO-Crate directories (which have been zipped) - that's a bug. But the big one is the temporary git clones because the "main" git checkout is locked. This happens when someone loads a large multi-GB repository and gets tired of it being slow and click Refresh - basically they are then causing a concurrency conflict against themselves..

Michael R. Crusoe

@mr-c

yikes, maybe Curii can add a lock around that second case

Stian Soiland-Reyes

@stain

yeah, get some kind of better workaround.. perhaps the locks are rather cleared or just an error message. Perhaps the lock should expire.

I guess the intention was a "continue to protect and serve" but at the cost of disk/network usage

Michael R. Crusoe

@mr-c

I made an issue common-workflow-language/cwlviewer#279

Stian Soiland-Reyes

@stain

FYI: we just had a call with Curii about them taking over hosting of CWL Viewer onto a new AWS instance instead of the current University of Manchester machine. Hopefully fix some of the issues on /tmp mentioned above.

Where communities thrive

People

Repo info

Activity