Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
ardydavari
@ardydavari
This may sound like a strange request--but I've been wondering if it would be possible to install dockstore binary in a container, and have that container orchestrate calls to its dependencies (i.e through docker-compose)?
Denis Yuen
@denis-yuen
@ardydavari I don't think it is impossible, but I don't think it would be straight-forward and would involve some hacking, mostly due to the trickiness of using docker in docker.
ardydavari
@ardydavari
@denis-yuen Maybe as long as the docker port is exposed in the dockstore container, this could be possible without docker in docker i.e. docker run -v /var/run/docker.sock:/var/run/docker.sock ...
ardydavari
@ardydavari
During various parts of the execution of my variant calling pipelines, I am getting this error regarding tar "file changed as we read it". Has anyone seen this before? I am thinking it might be related to the intermediate cache folder being rewritten during execution. I wonder if I actually moved the docker data-root correctly.
Konrad Rokicki
@krokicki
Hi, I'm new to this ecosystem, so my apologies if this is obvious or previously answered somewhere. I love the idea of describing tools using containers and descriptors, but would prefer to avoid additional service dependencies. Have there been any thoughts about creating "self-contained" tools which are self-describing? For example, containerized tools which have descriptors (e.g. CWL) embedded inside the container itself, and which can be interrogated about their capabilities without depending on a service like Dockstore?
Michael R. Crusoe
@mr-c
@krokicki Welcome! There is a canonical filesystem location for CWL descriptions:/usr/share/cwl/
Likewise, a CWL description can point to a software container, if you want to go that direction
Konrad Rokicki
@krokicki
Thanks, @mr-c , that sounds perfect! Is that path in any official spec/standard? It seems pretty trivial to copy the CWL into the container during the build, but I don't see anyone actually doing it (e.g. when searching GitHub for that path). I did find this spec: https://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem but it suggests a different path: /usr/share/commonwl/
@krokicki the spec is authoritative, I think @mr-c mistyped
@krokicki we could specify an entry point like /usr/share/commonwl/main.cwl but you'll want an easy way to extract a file or directory from a container image at rest, I think docker cp only works on live containers
Peter Amstutz
@tetron
@krokicki another idea was to shove the CWL into a metadata label which seems slightly abusive of the label system but is relatively easy to inspect
Michael R. Crusoe
@mr-c
Pardon me, it is /usr/share/commonwl/as @tetron pointed out
Konrad Rokicki
@krokicki
@tetron Thanks, that was a useful discussion to read. I think we can extract without running the container, e.g. https://unix.stackexchange.com/questions/331645/extract-file-from-docker-image
Peter Amstutz
@tetron
@krokicki looks like docker create still extracts the image to create a root fs
but there's probably a tool somewhere to do this efficiently
Konrad Rokicki
@krokicki
The way I imagine it working is that someone pastes a docker registry URI into our workflow tool, and our tool downloads the image and extracts the CWL and caches it. This would only happen rarely, so doing it efficiently wouldn't be a huge concern. But if there's a better way it certainly wouldn't hurt.
Peter Amstutz
@tetron
@all weekly CWL video chat is happening in five minutes https://meet.jit.si/cwl
I'm sorry wrong channel
@krokicki this would be a good conversation for https://gitter.im/common-workflow-language/common-workflow-language and the CWL video chat (happening shortly)
Konrad Rokicki
@krokicki
Thanks, I will follow up there next.
Peter Amstutz
@tetron
@denis-yuen I don't know if you're aware, the CWL community has started collecting tools at https://github.com/common-workflow-library/bio-cwl-tools and wanted to suggest that Dockstore could scrape that repository to make it searchable
Denis Yuen
@denis-yuen
I wasn't aware, thanks for the heads-up though. Will create a ticket to hook it up
Peter Amstutz
@tetron
awesome
we were talking about how we didn't really want to build our own index, but instead get existing sites like dockstore and bio.tools to include it
Denis Yuen
@denis-yuen
Yeah, I think it makes sense and it will be a good addition to Dockstore.
Peter Amstutz
@tetron
tools that get added to that repository have to pass some QA checks so if there is any QA checks or metadata that would be helpful for Dockstore import you should suggest it
Denis Yuen
@denis-yuen
I see file formats in https://github.com/common-workflow-library/bio-cwl-tools/blob/release/kraken2/kraken2.cwl , that would be awesome as a "recommended" or "stretch" goal
Michael R. Crusoe
@mr-c
@denis-yuen I agree! Can you send a PR to add that to https://github.com/common-workflow-library/bio-cwl-tools#styleguide ?
1 reply
Keiran Raine
@keiranmraine
Hi @denis-yuen , is there a guide for defining workflows in dockstore that link to a shared repository?
Denis Yuen
@denis-yuen

@keiranmraine
Short answer: not yet, it might work best with automated tools
Long answer: we're working on it starting with workflows (i.e. class: Workflow) in conjunction with GitHub apps. That's a new feature detailed in https://docs.dockstore.org/en/develop/getting-started/github-apps/github-apps.html that also allows for a much easier way to define multiple workflows in one repo. We also have a feature in the pipe that deals with one of the issues of workflows in a shared repo (tags applies to all tools regardless of whether they changed)

We have not started thinking about GitHub apps with tools (class: CommandLineTool) yet but it is on our roadmap.
In the meantime, if you register tools as an automated build via quay.io that might be the cleanest method ... but I do see that https://github.com/common-workflow-library/bio-cwl-tools doesn't have Dockerfiles (so that might not work either)

dockstore/dockstore#3695 if curious about the filters issue
Michael R. Crusoe
@mr-c
@denis-yuen Almost all of those tool descriptions use existing biocontainers.pro containers (which is a feature and a policy decision)
Denis Yuen
@denis-yuen
@mr-c do you happen to know if the biocontainers.pro containers that are hosted on quay.io are built on quay.io or built elsewhere and pushed to quay.io?
Denis Yuen
@denis-yuen
Also while asking, I should also ask what is the branching and tag plan for this repo? It will help our planning
e.g. do you forsee frequent updates and tags targeted at specific tools? less frequent and larger roll-up tags for the repo as a whole?
Keiran Raine
@keiranmraine
Hi, I had an issue with the cli and could do with confirming I'm not doing something silly: dockstore/dockstore#3784
Keiran Raine
@keiranmraine
Happy Labor day BTW
Denis Yuen
@denis-yuen
@keiranmraine took a brief look at it at https://github.com/dockstore/dockstore/issues/3784#issuecomment-688912578
Thanks for pointing it out, I think we missed the ticket. I don't see something obvious at first glance, but it does indeed seem like something related to our understanding of the parameter types when rewriting the input json from remote data to local data after downloading
Kaushik Ghose
@kaushik-work
Folks, I'm trying to figure out how to get the trs:// URI for a Dockstore workflow. When I go to the "TRS" line in the workflow summary page, the link is an https:// link. While cwltool works fine with that, it is not a trs:// link as outlined in the specification. I can not figure out what the TRS link should be. It would be nice to have that as the text on the workflow page, instead of the stub that we have now.
So, instead of #workflow/github.com/kaushik-work/simple-passthrough-cwl it would be helpful to have trs://dockstore.org/workflow/github.com/kaushik-work/simple-passthrough-cwl or whatever the proper URI is.
Denis Yuen
@denis-yuen
I think it would be trs://dockstore.org/#workflow/github.com/kaushik-work/simple-passthrough-cwl based on the pattern trs://<server>/<id>
But to the larger point, you're right. We display the ID but not a link of that format yet. Will create a ticket dockstore/dockstore#3823
Kaushik Ghose
@kaushik-work
Thanks @denis-yuen !
pvanheus
@pvanheus
What qualifies as a "tool" in the Dockstore search? This search - https://dockstore.org/search?descriptorType=CWL&_type=tool&searchMode=files - seems to show quite a few workflows.
Michael R. Crusoe
@mr-c
@pvanheus any combination of a CWL + a single container?
Asking on https://discuss.dockstore.org/ should yield a definitive answer
Denis Yuen
@denis-yuen
@pvanheus for CWL, CommandLineTools are "tools", workflows are well workflows. That chart that @mr-c linked shows the criteria that we use for WDL which is a bit more artificial
pvanheus
@pvanheus
ah ok, I see some of these are workflows in other languages (I saw Perl, for example) with CWL wrappers. So yes CommandLineTool formally :)
1 reply