Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Dominic Fitzgerald
@djf604
It looks like it's trying to run this command "['docker', 'pull', 'quay.io/wtsicgp/dockstore-cgpwgs:1.1.4']"
Andrew Duncan
@agduncan94
Could you try running cwltool directly? I just want to confirm that this is with the Dockstore portion of the code. You can run
"cwltool Docsktore.cwl example.json"
Dominic Fitzgerald
@djf604
Same error, it's trying to run docker pull
There is this line, if it's important: "Dockstore.cwl:18:1: unrecognized extension field dct:creator. Did you include a $schemas section?"
Andrew Duncan
@agduncan94
With dockstore when we run cwltool, we set the --non-strict so that the error doesn't show up. If that is from running cwltool directly I think it should be fine.
For the docker pull, cwltool has an option "--force-docker-pull Pull latest docker image even if it is locally present", which leads me to believe that by default it will use a local image if it is present.
There is a --disable-pull option
"--disable-pull Do not try to pull Docker images"
So maybe
"cwltool Dockstore.cwl example.json --disable-pull"
Dominic Fitzgerald
@djf604
Putting --disable-pull on the end threw an argument error, and putting it right after cwltool threw the same error as before
Andrew Duncan
@agduncan94
Can you confirm that right image and version are there with
"docker images"
Dominic Fitzgerald
@djf604
Well I see this on the head node: "quay.io/wtsicgp/dockstore-cgpwgs 1.1.4 8f2d9de5ab9d 2 months ago 1.56GB"
And this on the compute node: "<none> <none> 8f2d9de5ab9d 2 months ago 1.56GB"
The image id is the same, but does it matter that on the compute node the repository and tag are missing?
Andrew Duncan
@agduncan94
I think so :(
Dominic Fitzgerald
@djf604
Ok I'll see what I can do about populating it
I really appreciate your help here @agduncan94 , I brought cookies into work today if you want one and are willing to travel to Chicago haha
Andrew Duncan
@agduncan94
No problem, hopefully we can figure something out that works for you.
haha I’ll hop on the next flight
Dominic Fitzgerald
@djf604
Yep that was it. Once the repo and tag names were populated in "docker images" then the whole thing runs offline
Andrew Duncan
@agduncan94
Awesome :smile:
I've created an issue ga4gh/dockstore#1854
We will likely add the solution to our documentation site, since I'm sure in the future more people will have a similar use case.
How did you manage to populate the repo and tag names?
Dominic Fitzgerald
@djf604
My coworker did it, let me find out
Dominic Fitzgerald
@djf604
Ok just to re-explain my situation, I was able to run the dockstore container just fine on the head node of our commodity cluster, which has internet access. Of course the point of having a cluster is to run software on compute nodes, which do not have public internet access. So when trying to execute the same command on a compute node it was complaining that it couldn't pull down the docker image
So we saved the docker image with "docker save -o dockstore-cgpwgs_with_tags quay.io/wtsicgp/dockstore-cgpwgs:1.1.4" (executed on the head node to some cluster-globally-accessible directory), then added it to the compute node(s) with "docker load -i /path/to/image.tar" (executed on the compute node)
What we had done before, which caused the lack of entries in the repo and tag column was "docker save -o dockstore-cgpwgs.tar 8f2d9de5ab9d", so just giving the tag id doesn't place nicely with dockstore
I did also have to do the whole "dockstore download" business, but that was an easy step that I'm sure is well documented for you
After that, the command "dockstore tool launch --local-entry Dockstore.cwl --json example.json" worked just fine on our offline compute node
Is that helpful?
Andrew Duncan
@agduncan94
this is perfect, thank you
Dominic Fitzgerald
@djf604
No thank you so much. Your help today opened up a weeks long bottleneck, it's much appreciated!
Keiran Raine
@keiranmraine
The onboarding page is not very easy to access once you have an account anymore unless you know the url. Is this intentional or am I blind?
Andrew Duncan
@agduncan94
There is a link on the homepage called setup account, but yeah we were discussing how we can make it more visible after initial onboarding.
The link external accounts is visible in the accounts page, and same with the change username.
Keiran Raine
@keiranmraine
Ah, it used to be under the account area once logged in. Now I've found the API curl I'm good... although I've noticed that if you specify python3 the cwl version downgrades significantly
Gary Luu
@garyluu
ya, i haven't tested with python3 very much, so didn't update the cwl version as frequent
Keiran Raine
@keiranmraine
$ diff -y requirements_py2.txt requirements_py3.txt 
setuptools==36.5.0                        setuptools==36.5.0
cwl-runner                            cwl-runner
cwltool==1.0.20180403145700                      |    cwltool==1.0.20170828135420
schema-salad==2.7.20180514132321                  |    schema-salad==2.6.20170806163416
avro==1.8.1                              |    avro-cwl==1.8.3
ruamel.yaml==0.14.12                        ruamel.yaml==0.14.12
requests==2.18.4                        requests==2.18.4
Andrew Duncan
@agduncan94
Oh yes so we had it visible up until users perform certain actions which imply they have a decent understanding of Dockstore (such as publish a tool), and then we hide it once they perform any of those actions.
Keiran Raine
@keiranmraine
Ah, okay good to know. We need the 2018 features for monitoring docker containers
Andrew Duncan
@agduncan94
We are going to add a link to the onboarding on the accounts page as well in the next hotfix I believe
Also we will create an issue for testing with python 3
Denis Yuen
@denis-yuen
@keiranmraine do you have a link to the 2018 features that you're referring to?
Keiran Raine
@keiranmraine
Options for recording the Docker container identifier into a file:
  --record-container-id
                        If enabled, store the Docker container ID into a file.
                        See --cidfile-dir to specify the directory.
  --cidfile-dir CIDFILE_DIR
                        Directory for storing the Docker container ID file.
                        The default is the current directory
  --cidfile-prefix CIDFILE_PREFIX
                        Specify a prefix to the container ID filename. Final
                        file name will be followed by a timestamp. The default
                        is no prefix.
Sorry @denis-yuen , we want to be able to point some monitoring tools to the correct containers via the cidfiles
Michael R. Crusoe
@mr-c
@keiranmraine you can use cwltool directly, it speaks GA4GH TRS to dockstore.org https://github.com/common-workflow-language/cwltool#use-with-ga4gh-tool-registry-api
Keiran Raine
@keiranmraine
Hi @mr-c, we are using it directly in many cases. I raised it as we build our base image using the requirements.txt from dockstore (as it's via a REST call) so people can choose how they interact.
(I've raised a small, unrelated, Q on cwl gitter channel too)
Keiran Raine
@keiranmraine
@denis-yuen @agduncan94 cwl-runner complains about the absence of html5lib (py2), would be good to get it into the requirements REST route.