Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Konrad Rokicki
@krokicki
@tetron Thanks, that was a useful discussion to read. I think we can extract without running the container, e.g. https://unix.stackexchange.com/questions/331645/extract-file-from-docker-image
Peter Amstutz
@tetron
@krokicki looks like docker create still extracts the image to create a root fs
but there's probably a tool somewhere to do this efficiently
Konrad Rokicki
@krokicki
The way I imagine it working is that someone pastes a docker registry URI into our workflow tool, and our tool downloads the image and extracts the CWL and caches it. This would only happen rarely, so doing it efficiently wouldn't be a huge concern. But if there's a better way it certainly wouldn't hurt.
Peter Amstutz
@tetron
@all weekly CWL video chat is happening in five minutes https://meet.jit.si/cwl
I'm sorry wrong channel
@krokicki this would be a good conversation for https://gitter.im/common-workflow-language/common-workflow-language and the CWL video chat (happening shortly)
Konrad Rokicki
@krokicki
Thanks, I will follow up there next.
Peter Amstutz
@tetron
@denis-yuen I don't know if you're aware, the CWL community has started collecting tools at https://github.com/common-workflow-library/bio-cwl-tools and wanted to suggest that Dockstore could scrape that repository to make it searchable
Denis Yuen
@denis-yuen
I wasn't aware, thanks for the heads-up though. Will create a ticket to hook it up
Peter Amstutz
@tetron
awesome
we were talking about how we didn't really want to build our own index, but instead get existing sites like dockstore and bio.tools to include it
Denis Yuen
@denis-yuen
Yeah, I think it makes sense and it will be a good addition to Dockstore.
Peter Amstutz
@tetron
tools that get added to that repository have to pass some QA checks so if there is any QA checks or metadata that would be helpful for Dockstore import you should suggest it
Denis Yuen
@denis-yuen
I see file formats in https://github.com/common-workflow-library/bio-cwl-tools/blob/release/kraken2/kraken2.cwl , that would be awesome as a "recommended" or "stretch" goal
Michael R. Crusoe
@mr-c
@denis-yuen I agree! Can you send a PR to add that to https://github.com/common-workflow-library/bio-cwl-tools#styleguide ?
1 reply
Keiran Raine
@keiranmraine
Hi @denis-yuen , is there a guide for defining workflows in dockstore that link to a shared repository?
Denis Yuen
@denis-yuen

@keiranmraine
Short answer: not yet, it might work best with automated tools
Long answer: we're working on it starting with workflows (i.e. class: Workflow) in conjunction with GitHub apps. That's a new feature detailed in https://docs.dockstore.org/en/develop/getting-started/github-apps/github-apps.html that also allows for a much easier way to define multiple workflows in one repo. We also have a feature in the pipe that deals with one of the issues of workflows in a shared repo (tags applies to all tools regardless of whether they changed)

We have not started thinking about GitHub apps with tools (class: CommandLineTool) yet but it is on our roadmap.
In the meantime, if you register tools as an automated build via quay.io that might be the cleanest method ... but I do see that https://github.com/common-workflow-library/bio-cwl-tools doesn't have Dockerfiles (so that might not work either)

dockstore/dockstore#3695 if curious about the filters issue
Michael R. Crusoe
@mr-c
@denis-yuen Almost all of those tool descriptions use existing biocontainers.pro containers (which is a feature and a policy decision)
Denis Yuen
@denis-yuen
@mr-c do you happen to know if the biocontainers.pro containers that are hosted on quay.io are built on quay.io or built elsewhere and pushed to quay.io?
Denis Yuen
@denis-yuen
Also while asking, I should also ask what is the branching and tag plan for this repo? It will help our planning
e.g. do you forsee frequent updates and tags targeted at specific tools? less frequent and larger roll-up tags for the repo as a whole?
Keiran Raine
@keiranmraine
Hi, I had an issue with the cli and could do with confirming I'm not doing something silly: dockstore/dockstore#3784
Keiran Raine
@keiranmraine
Happy Labor day BTW
Denis Yuen
@denis-yuen
@keiranmraine took a brief look at it at https://github.com/dockstore/dockstore/issues/3784#issuecomment-688912578
Thanks for pointing it out, I think we missed the ticket. I don't see something obvious at first glance, but it does indeed seem like something related to our understanding of the parameter types when rewriting the input json from remote data to local data after downloading
Kaushik Ghose
@kaushik-work
Folks, I'm trying to figure out how to get the trs:// URI for a Dockstore workflow. When I go to the "TRS" line in the workflow summary page, the link is an https:// link. While cwltool works fine with that, it is not a trs:// link as outlined in the specification. I can not figure out what the TRS link should be. It would be nice to have that as the text on the workflow page, instead of the stub that we have now.
So, instead of #workflow/github.com/kaushik-work/simple-passthrough-cwl it would be helpful to have trs://dockstore.org/workflow/github.com/kaushik-work/simple-passthrough-cwl or whatever the proper URI is.
Denis Yuen
@denis-yuen
I think it would be trs://dockstore.org/#workflow/github.com/kaushik-work/simple-passthrough-cwl based on the pattern trs://<server>/<id>
But to the larger point, you're right. We display the ID but not a link of that format yet. Will create a ticket dockstore/dockstore#3823
Kaushik Ghose
@kaushik-work
Thanks @denis-yuen !
pvanheus
@pvanheus
What qualifies as a "tool" in the Dockstore search? This search - https://dockstore.org/search?descriptorType=CWL&_type=tool&searchMode=files - seems to show quite a few workflows.
Michael R. Crusoe
@mr-c
@pvanheus any combination of a CWL + a single container?
Asking on https://discuss.dockstore.org/ should yield a definitive answer
Denis Yuen
@denis-yuen
@pvanheus for CWL, CommandLineTools are "tools", workflows are well workflows. That chart that @mr-c linked shows the criteria that we use for WDL which is a bit more artificial
pvanheus
@pvanheus
ah ok, I see some of these are workflows in other languages (I saw Perl, for example) with CWL wrappers. So yes CommandLineTool formally :)
1 reply
Keiran Raine
@keiranmraine
I suspect this may be of interest:

Me:

I don't seem to be able to find any documentation or settings that allow you to set an account/password to allow pulls from docker hub via a docker login for base images.

Although I've read the suggested mitigation (https://www.openshift.com/blog/mitigate-impact-of-docker-hub-pull-request-limits [openshift.com]) this doesn't appear to address sensible methods to deal with things like "ubuntu:20.04". It would seem incredibly inefficient for each group needing an OS image to replicate this on quay.io under their own repo.

Is quay.io going to provide tooling for this our do we need to move to all images being built by an external CI (pulling from dockerhub with login) and being pushed to quay (via docker login)?

Quay.io

Quay does not have a mechanism to set up pull credentials for images on outside registries. It expects that the image you pull is publicly available. The only way to mitigate this issue is to move your base images to a free and open registry that does not rate limit the number of pulls.

I have created a feature request for our dev team that would allow users to define pull credentials at build time. That way you could add Docker credentials to your build which should make your pull authenticated.

Unfortunately, Ican't tell you when this feture will be implemented, I did create it with high priority as this issue hits many of our customers.

Denis Yuen
@denis-yuen
@keiranmraine thanks for the heads-up. I'm actually a little surprised that they don't have the equivalent of a pull through cache or something https://docs.docker.com/registry/recipes/mirror/
Quay.io must be pulling a lot of data from Docker Hub
Keiran Raine
@keiranmraine
@denis-yuen apparently a local cache is still a hit, so I'm guessing a pull through would be no different. If you ask for "ubuntu:20.04" your mirror would still check the original source to see if the name references the same hash (due to security updates being applied).
Denis Yuen
@denis-yuen

It is important to note however, that a pull is also counted if the client system already has all the image layers present and nothing is actually downloaded. That means that image caching does not reduce the number of pulls counted against the limit.

confirmed, that indeed sucks

Keiran Raine
@keiranmraine
@denis-yuen my comminication with Ubuntu/Canonical has yielded:

Hi Keiran,

I manage the Sanger account at Canonical.

Today we announced that we will be joining the Docker Hub, verified publisher programme
https://www.docker.com/blog/canonical-joins-docker-verified-publisher-program
https://ubuntu.com/blog/canonical-publishes-lts-docker-image-portfolio-on-docker-hub
There are no rate limits because of this new partnership.

Specifically in the second link:

Canonical and Docker will collaborate on Docker Official Images and LTS Docker Image Portfolio to bring the best of the two to the community and ecosystem. The entire LTS Docker Image Portfolio will be exempted from per-user rate limits.

I expect other OS vendors will follow suit
Keiran Raine
@keiranmraine
FYI, expected some time next week
Michael R. Crusoe
@mr-c
This is good news, thank you @keiranmraine
Denis Yuen
@denis-yuen
:+1: