Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Peter Amstutz
@tetron
but yea they should be able to execute from Dockstore
although we (ga4gh) haven't officially integrated using trs:// and drs:// identifiers with WES
HashRocketSyntax
@HashRocketSyntax
has anyone tackled a k8s helm chart that (a) runs each of the backends, and (b) service for routing WES requests to those backends?
Peter Amstutz
@tetron
that sounds like something Susheel Varma at Elixer/EBI is interested in
now you're going to say either you are Susheel or you work for him :-)
HashRocketSyntax
@HashRocketSyntax
nah, im layne from nextcode + genomics ireland
we have hooked nextflow up to k8s to run stuff at wgs, cohort scale. it is working well... very well.
just thinking about how we do federated workflows at different sites
Chris Llanwarne
@cjllanwarne
hello folks! Is this the right place to find help registering an account? I'm getting webservice errors right now?
Denis Yuen
@denis-yuen
Yup, this is the right place. Does it happen to look like dockstore/dockstore#3008
Chris Llanwarne
@cjllanwarne

warning The webservice encountered an error.

[HTTP 500] OK: org.hibernate.exception.DataException: could not execute statement could not execute statement ERROR: value too long for type character varying(255)

Denis Yuen
@denis-yuen
ok yup
Chris Llanwarne
@cjllanwarne
oh, yeah, that looks exactly like it :smile:
so google login is not the way to go right now?
I can easily enough just use my github
Denis Yuen
@denis-yuen
I haven't had a chance to catch-up with Charles, but I would guess that unless its possible to shorten the avatarURL (is it configurable?), then yes
Chris Llanwarne
@cjllanwarne
what's the avatarURL? Is that something I can double check myself?
For example, Charles's is https://lh6.googleusercontent.com/-LNh1k32Tsyc/AAAAAAAAAAI/AAAAAAAAABg/u79rJaDzkvQ/photo.jpg which looks like 92 characters. We must be systematically hitting 256 characters, wonder if its a google apps/corporate thing
Denis Yuen
@denis-yuen
@cjllanwarne I think if you go to https://aboutme.google.com/u/0/?referer=gplus and open image with new tab on your avatar, you can see how long the URL is. I think that also means it isn't really configurable
Chris Llanwarne
@cjllanwarne
my url appears to be about 800 characters long, so yeah, more than 255
setting up with github appears to have worked though, so at least for now I'm unblocked
ardydavari
@ardydavari
Does anyone know the size of TMPDIR for the dockstore containers? Is there a maximum file size before things start to get rewritten?
Denis Yuen
@denis-yuen
@cjllanwarne thanks, added the info to the ticket
Chris Llanwarne
@cjllanwarne
thank you!
Denis Yuen
@denis-yuen
@ardydavari not offhand, I don't think we do anything special so it might be just a typical Docker question, possibly dependent on if you're using CWL/WDL and the workflow engine passes any special settings to the "docker run" command
ardydavari
@ardydavari
I think you guys are right. I think my real question is how to correctly to remap the default docker data location to another mountpoint (working on this now)
ardydavari
@ardydavari

This is probably beyond the scope of this forum, but does anyone know offhand if there are any additional parameters that need to be changed besides adding the following to /etc/docker/docker.json

{
        "data-root": "/data/docker-data"
}

The parameters above seem to remap /etc/hostname, /etc/hosts /etc/resolv.conf, appropriately except for the root folder which is still size limited to 10G

ardydavari
@ardydavari
I think I may have answered my own question. Adding "storage-opts": ["dm.basesize=100G"], increases the base image size.
ardydavari
@ardydavari
This may sound like a strange request--but I've been wondering if it would be possible to install dockstore binary in a container, and have that container orchestrate calls to its dependencies (i.e through docker-compose)?
Denis Yuen
@denis-yuen
@ardydavari I don't think it is impossible, but I don't think it would be straight-forward and would involve some hacking, mostly due to the trickiness of using docker in docker.
ardydavari
@ardydavari
@denis-yuen Maybe as long as the docker port is exposed in the dockstore container, this could be possible without docker in docker i.e. docker run -v /var/run/docker.sock:/var/run/docker.sock ...
ardydavari
@ardydavari
During various parts of the execution of my variant calling pipelines, I am getting this error regarding tar "file changed as we read it". Has anyone seen this before? I am thinking it might be related to the intermediate cache folder being rewritten during execution. I wonder if I actually moved the docker data-root correctly.
Konrad Rokicki
@krokicki
Hi, I'm new to this ecosystem, so my apologies if this is obvious or previously answered somewhere. I love the idea of describing tools using containers and descriptors, but would prefer to avoid additional service dependencies. Have there been any thoughts about creating "self-contained" tools which are self-describing? For example, containerized tools which have descriptors (e.g. CWL) embedded inside the container itself, and which can be interrogated about their capabilities without depending on a service like Dockstore?
Michael R. Crusoe
@mr-c
@krokicki Welcome! There is a canonical filesystem location for CWL descriptions:/usr/share/cwl/
Likewise, a CWL description can point to a software container, if you want to go that direction
Konrad Rokicki
@krokicki
Thanks, @mr-c , that sounds perfect! Is that path in any official spec/standard? It seems pretty trivial to copy the CWL into the container during the build, but I don't see anyone actually doing it (e.g. when searching GitHub for that path). I did find this spec: https://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem but it suggests a different path: /usr/share/commonwl/
@krokicki the spec is authoritative, I think @mr-c mistyped
@krokicki we could specify an entry point like /usr/share/commonwl/main.cwl but you'll want an easy way to extract a file or directory from a container image at rest, I think docker cp only works on live containers
Peter Amstutz
@tetron
@krokicki another idea was to shove the CWL into a metadata label which seems slightly abusive of the label system but is relatively easy to inspect
Michael R. Crusoe
@mr-c
Pardon me, it is /usr/share/commonwl/as @tetron pointed out
Konrad Rokicki
@krokicki
@tetron Thanks, that was a useful discussion to read. I think we can extract without running the container, e.g. https://unix.stackexchange.com/questions/331645/extract-file-from-docker-image
Peter Amstutz
@tetron
@krokicki looks like docker create still extracts the image to create a root fs
but there's probably a tool somewhere to do this efficiently
Konrad Rokicki
@krokicki
The way I imagine it working is that someone pastes a docker registry URI into our workflow tool, and our tool downloads the image and extracts the CWL and caches it. This would only happen rarely, so doing it efficiently wouldn't be a huge concern. But if there's a better way it certainly wouldn't hurt.
Peter Amstutz
@tetron
@all weekly CWL video chat is happening in five minutes https://meet.jit.si/cwl
I'm sorry wrong channel
@krokicki this would be a good conversation for https://gitter.im/common-workflow-language/common-workflow-language and the CWL video chat (happening shortly)
Konrad Rokicki
@krokicki
Thanks, I will follow up there next.