Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Ben
@BenjaminHCCarr
that would explain the high failure rate then
Michael R. Crusoe
@mr-c
😁
Ben
@BenjaminHCCarr
Michael R. Crusoe
@mr-c
Yay
Stian Soiland-Reyes
@stain

@BenjaminHCCarr it's sane to submit all of them to the queue - but I think it does do git checkout on submission (to check the file path) so it might still be slow. Here's how I do restore from backup:
https://github.com/common-workflow-language/cwlviewer/blob/1.3.0-RC1/load.py#L76

as you see I let it have 6 at a time, sleeping 0.5s to let queue shrink a bit

that one works fine for re-loading all ~4190 workflows
Michael R. Crusoe
@mr-c
I have submitted the "CWL community" as a service provider to the EOSC Portal so that we may advertise the CWL Viewer itself: https://eosc-portal.eu/for-providers
Michael R. Crusoe
@mr-c
Interesting, the CWL Viewer has over 100 000 downloads of the docker container, more than any other CWL organization container on the Docker Hub: https://hub.docker.com/u/commonworkflowlanguage
Peter Amstutz
@tetron
Huh, that many downloads sounds like it is being pulled by some CI infrastructure
Sounds like there could be a project to make some part of cwlviewer into a standalone tool
I think that came up at the Meetup a couple weeks ago
Chris Hunter
@only1chunts
Hi, not sure if this is the right place to bring this up, but i hope so!? I'm interested in embedding the interactive workflow image(s) within (publishable) dataset webpages, for us, the simpliest way to do that would be to add them as a widget view. Does the CWL-viewer tool already have this, or would it be a feature you could add? By way of background, I'm looking to include author generated workflows within GigaDB datasets, we would like to do this in a similar way to how we include SketchFab interactive 3D models, and protocols.io. i.e. we just add the widget code to a dataset and the it displays the relevant item in a window within the dataset page. Here is a link to a SketchFab example; open the DOI http://dx.doi.org/10.5524/100539 navigate to the tab labelled "3D viewer" . We would like to add a tab in the framework for "CWL viewer" to allow those datasets that have it to display the interactive workflow there. Any comments? Thanks.
Michael R. Crusoe
@mr-c
Welcome @only1chunts This is a good place to start the conversation.
Are these workflows stored on GitHub, GitLab, or some other publicly accessible Git repository? If so then you can use the API: https://view.commonwl.org/apidocs
Chris Hunter
@only1chunts
The example of the sketchfab widget does not have a workflow, it was just to demonstrate how I would like to be able to integrate external sources such as CWL-viewer. The most recent dataset we have published with a workflow (in GitHub) is one that has already been put through cwl-viewer, the only way I can link to it at the moment from GigaDB is just as a URL for users to click, see here http://dx.doi.org/10.5524/100553
Peter Amstutz
@tetron
@only1chunts there is cwl-svg from the rabix project, that might be better for embedding
Michael R. Crusoe
@mr-c
@only1chunts Thanks for linking to the CWL Viewer! The .cwl is missing from the link, here's the correct one: https://view.commonwl.org/workflows/github.com/vetscience/Assemblosis/blob/66660a64824e4c1019680c6c7ea489ea359cd5f6/Run/assembly.cwl
@only1chunts You could put that URL in an iframe? Oh, and you should use the permalink version instead of the view.commonwl.org version: https://w3id.org/cwl/view/git/66660a64824e4c1019680c6c7ea489ea359cd5f6/Run/assembly.cwl
Stian Soiland-Reyes
@stain
Hi, @only1chunts - you can use the SVG directly, I am not sure what interactivity you want, the same zoom-buttons, selection etc. as on the CWL Viewer page?
but if the workflow is not in a git repository it's a bit more tricky because of versioning and relative links. In theory it will work straight on the web (we used to do that) but now we always checkout from git in a way to force people to do the "right thing"
@only1chunts - are you perhaps considering storing the CWL definitions also within GigaDB? We would need them to be directly on the web (no landing pages) and either all-in-one-file (which is not common in CWL community as people like to split out their tools) or with neighbouring CWL files available from the same base URI. If GitHub/GitLab is tricky for you to require, perhaps some in-between solution would be a downloadable .git zip archive which would be easier for us to integrate? Harder for users to upload to you though.
Stian Soiland-Reyes
@stain

@mr-c

stain@heater:~$ crontab -e
*/6 * * * * /home/stain/cwlviewer-dev/update.sh

It's the update of the dev instance from master that is responsible for most of those 100.000 pulls.. I've reduced it down to once an hour.

Michael R. Crusoe
@mr-c
@stain Ah-ha! Good to know :-)
Chris Hunter
@only1chunts
@mr-c thanks for the pointer about using the permalink, i thought i had! but obvious just cut and paste the wrong link. I'll change that now, it will also reduce the length which is why the .cwl was missing, (our database table truncated it). The iFrame is exactly what we want to do, but would like it to point to a place that has been specifically designed to view in a smaller frame (i.e. cut down to remove surround stuff like the the full list of tables below the interactive viewer is not required for a simple iframe view, If I can workout how to append an image I'll show you the view that we would like to see from within a GigaDB dataset , but if you dont feel the need for it then we could iframe the entire page within GigaDB.
I cant see how to add an image so I'll email it to you @mr-c
Chris Hunter
@only1chunts
Hi @stain , sorry I perhaps have not explained my use case very well. We are not (at this point) concerned with how an author /user gets their workflow in CWL viewer, just how we can present it IF they have done so. A static SVG image is an option, but its not great to look at when its a complex workflow where the boxes are tiny, the zoom functions in the viewer are very useful then. Plus, I'm assuming at some point in the future the viewer will have more interactivity added for links to tools and intersecting workflows etc, which would then be lost in the SVG version. Thanks.
Stian Soiland-Reyes
@stain
ah, so you would prefer some kind of 'embed' view that avoids the footer/header really?
I think that could be doable by a small change to the templates
Stian Soiland-Reyes
@stain
https://github.com/common-workflow-language/cwlviewer/blob/master/src/main/resources/templates/workflow.html is the normal workflow page template, note that it has the SVG viewer twice as there is also the modal pop-up. Perhaps adding ?embed=trueto the permalinks would be an option? It already works with ?format=html although that gives the big page now https://w3id.org/cwl/view/git/f55f0011de7db66f873ffb0f700f3fe26b33911e/genomel/cwl/workflows/harmonization/harmonization_bwa_mem.cwl?format=html
Michael R. Crusoe
@mr-c
I think the embed should still have the footer (or a version of it)
Should the steps, inputs & output be included? Or should they be replaced with a link to the "full" view?
Chris Hunter
@only1chunts
Thanks for thinking I might be a developer, but sadly you've just gone way past my knowledge on html (or programming of any sort). If developing a widget viewer is not something anyone else needs/wants and its not on your todo list then thats fine, we'll just have to stick with the URL link for now.
Michael R. Crusoe
@mr-c
@only1chunts You can load the URL into an iframe the same way the other widget works today. Later we'll add an ?embed=true mode that is better formatted
Chris Hunter
@only1chunts
perfect, thanks
Stian Soiland-Reyes
@stain
Would something like cwl|valid also be cool on the CWL Viewer?
Is it too late to make GSOC2019 suggestions for CWL, @mr-c ?
Michael R. Crusoe
@mr-c
@stain nope, but needs to get up asap to attract student talent
Michael R. Crusoe
@mr-c
@stain Also, that badge looks great!
@stain time to reboot?
Error: cwltool version failed to run on this workflow:
! Cwltool ran successfully, but an unexpected error occurred in CWLViewer!
Help us by reporting it on Gitter or a Github issue
Stian Soiland-Reyes
@stain

hmm.. I get the same error as with cwltool --validate really.
Viewer:

Tool definition failed validation:
../data/git/aa420273539ad7bcf384c37614f6f636810042f2/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl:170:16: Workflow step output 'vcf_tbi' does not correspond to
../data/git/aa420273539ad7bcf384c37614f6f636810042f2/Tools/gatk3-HaplotypeCaller-XCORE1-ploidy1.cwl:51:1:   tool output (expected 'vcf', 'log')

Command line:

(base) stain@biggie:~$ cwltool --validate https://raw.githubusercontent.com/ddbj/human-reseq/master/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl
/home/stain/miniconda3/bin/cwltool 1.0.20190228200117
Tool definition failed validation:
https://raw.githubusercontent.com/ddbj/human-reseq/master/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl:170:16: Workflow step output 'vcf_tbi' does not correspond to
https://raw.githubusercontent.com/ddbj/human-reseq/master/Tools/gatk3-HaplotypeCaller-XCORE1-ploidy1.cwl:51:1:   tool output (expected 'vcf', 'log')
btw, that badge is just https://img.shields.io/badge/cwl-valid-B5314C.svg?logo=common-workflow-language but the viewer would need to have an in-between resource that checks it live.
Michael R. Crusoe
@mr-c
@stain both shields.io and badget.net support json endpoints
(shields is thought to have timeout issues; badgen is said to be faster in rendering)
Stian Soiland-Reyes
@stain
I suggested that as well on https://github.com/OBF/GSoC/pull/44/files (feel free to modulate)
Michael R. Crusoe
@mr-c
@stain I think this was the commit making the problem I reported earlier? But now it works! https://github.com/ddbj/human-reseq/blob/9064c301d858a796f8340d1e762e5916fd199da0/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl
Stian Soiland-Reyes
@stain
cwltool --validate https://raw.githubusercontent.com/ddbj/human-reseq/9064c301d858a796f8340d1e762e5916fd199da0/Workflows/bams2gvcf.woBQSR_male_chrXY_wXTR.cwl gives similar validation error, but this time about secondaryFiles - so in a way it should not have worked! ;)
Michael R. Crusoe
@mr-c
A CWL v1.0.x to CWL v1.1 standalone converter common-workflow-language/cwl-upgrader#26