Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Andrew Duncan
@agduncan94
This looks promising
Dominic Fitzgerald
@djf604
I have done that, I can see the image locally with "docker images", but how then do I get dockstore to actually use it instead of trying to pull it?
Andrew Duncan
@agduncan94
I'm going to try an experiment locally to see if I can get it working offline.
I'll let you know what I find
Dominic Fitzgerald
@djf604
Much appreciated @agduncan94
Andrew Duncan
@agduncan94
So I tried building an image locally and then disconnecting internet. Then I tried running my tool again and it still worked. So you should be able to launch offline using the command you mentioned above
"dockstore tool launch --local-entry Dockstore.cwl --json example.json"
Let me know if it works for you
Dominic Fitzgerald
@djf604
That does not work, it throws the error "Error response from daemon: Get https://quay.io/v2/: dial tcp: lookup quay.io on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address"
It looks like it's trying to run this command "['docker', 'pull', 'quay.io/wtsicgp/dockstore-cgpwgs:1.1.4']"
Andrew Duncan
@agduncan94
Could you try running cwltool directly? I just want to confirm that this is with the Dockstore portion of the code. You can run
"cwltool Docsktore.cwl example.json"
Dominic Fitzgerald
@djf604
Same error, it's trying to run docker pull
There is this line, if it's important: "Dockstore.cwl:18:1: unrecognized extension field dct:creator. Did you include a $schemas section?"
Andrew Duncan
@agduncan94
With dockstore when we run cwltool, we set the --non-strict so that the error doesn't show up. If that is from running cwltool directly I think it should be fine.
For the docker pull, cwltool has an option "--force-docker-pull Pull latest docker image even if it is locally present", which leads me to believe that by default it will use a local image if it is present.
There is a --disable-pull option
"--disable-pull Do not try to pull Docker images"
So maybe
"cwltool Dockstore.cwl example.json --disable-pull"
Dominic Fitzgerald
@djf604
Putting --disable-pull on the end threw an argument error, and putting it right after cwltool threw the same error as before
Andrew Duncan
@agduncan94
Can you confirm that right image and version are there with
"docker images"
Dominic Fitzgerald
@djf604
Well I see this on the head node: "quay.io/wtsicgp/dockstore-cgpwgs 1.1.4 8f2d9de5ab9d 2 months ago 1.56GB"
And this on the compute node: "<none> <none> 8f2d9de5ab9d 2 months ago 1.56GB"
The image id is the same, but does it matter that on the compute node the repository and tag are missing?
Andrew Duncan
@agduncan94
I think so :(
Dominic Fitzgerald
@djf604
Ok I'll see what I can do about populating it
I really appreciate your help here @agduncan94 , I brought cookies into work today if you want one and are willing to travel to Chicago haha
Andrew Duncan
@agduncan94
No problem, hopefully we can figure something out that works for you.
haha I’ll hop on the next flight
Dominic Fitzgerald
@djf604
Yep that was it. Once the repo and tag names were populated in "docker images" then the whole thing runs offline
Andrew Duncan
@agduncan94
Awesome :smile:
I've created an issue ga4gh/dockstore#1854
We will likely add the solution to our documentation site, since I'm sure in the future more people will have a similar use case.
How did you manage to populate the repo and tag names?
Dominic Fitzgerald
@djf604
My coworker did it, let me find out
Dominic Fitzgerald
@djf604
Ok just to re-explain my situation, I was able to run the dockstore container just fine on the head node of our commodity cluster, which has internet access. Of course the point of having a cluster is to run software on compute nodes, which do not have public internet access. So when trying to execute the same command on a compute node it was complaining that it couldn't pull down the docker image
So we saved the docker image with "docker save -o dockstore-cgpwgs_with_tags quay.io/wtsicgp/dockstore-cgpwgs:1.1.4" (executed on the head node to some cluster-globally-accessible directory), then added it to the compute node(s) with "docker load -i /path/to/image.tar" (executed on the compute node)
What we had done before, which caused the lack of entries in the repo and tag column was "docker save -o dockstore-cgpwgs.tar 8f2d9de5ab9d", so just giving the tag id doesn't place nicely with dockstore
I did also have to do the whole "dockstore download" business, but that was an easy step that I'm sure is well documented for you
After that, the command "dockstore tool launch --local-entry Dockstore.cwl --json example.json" worked just fine on our offline compute node
Is that helpful?
Andrew Duncan
@agduncan94
this is perfect, thank you
Dominic Fitzgerald
@djf604
No thank you so much. Your help today opened up a weeks long bottleneck, it's much appreciated!
Keiran Raine
@keiranmraine
The onboarding page is not very easy to access once you have an account anymore unless you know the url. Is this intentional or am I blind?
Andrew Duncan
@agduncan94
There is a link on the homepage called setup account, but yeah we were discussing how we can make it more visible after initial onboarding.
The link external accounts is visible in the accounts page, and same with the change username.
Keiran Raine
@keiranmraine
Ah, it used to be under the account area once logged in. Now I've found the API curl I'm good... although I've noticed that if you specify python3 the cwl version downgrades significantly
Gary Luu
@garyluu
ya, i haven't tested with python3 very much, so didn't update the cwl version as frequent
Keiran Raine
@keiranmraine
$ diff -y requirements_py2.txt requirements_py3.txt 
setuptools==36.5.0                        setuptools==36.5.0
cwl-runner                            cwl-runner
cwltool==1.0.20180403145700                      |    cwltool==1.0.20170828135420
schema-salad==2.7.20180514132321                  |    schema-salad==2.6.20170806163416
avro==1.8.1                              |    avro-cwl==1.8.3
ruamel.yaml==0.14.12                        ruamel.yaml==0.14.12
requests==2.18.4                        requests==2.18.4
Andrew Duncan
@agduncan94
Oh yes so we had it visible up until users perform certain actions which imply they have a decent understanding of Dockstore (such as publish a tool), and then we hide it once they perform any of those actions.
Keiran Raine
@keiranmraine
Ah, okay good to know. We need the 2018 features for monitoring docker containers