I've created an issue ga4gh/dockstore#1854
We will likely add the solution to our documentation site, since I'm sure in the future more people will have a similar use case.
How did you manage to populate the repo and tag names?
Ok just to re-explain my situation, I was able to run the dockstore container just fine on the head node of our commodity cluster, which has internet access. Of course the point of having a cluster is to run software on compute nodes, which do not have public internet access. So when trying to execute the same command on a compute node it was complaining that it couldn't pull down the docker image
So we saved the docker image with "docker save -o dockstore-cgpwgs_with_tags quay.io/wtsicgp/dockstore-cgpwgs:1.1.4" (executed on the head node to some cluster-globally-accessible directory), then added it to the compute node(s) with "docker load -i /path/to/image.tar" (executed on the compute node)
What we had done before, which caused the lack of entries in the repo and tag column was "docker save -o dockstore-cgpwgs.tar 8f2d9de5ab9d", so just giving the tag id doesn't place nicely with dockstore
I did also have to do the whole "dockstore download" business, but that was an easy step that I'm sure is well documented for you
After that, the command "dockstore tool launch --local-entry Dockstore.cwl --json example.json" worked just fine on our offline compute node
Is that helpful?
The link external accounts is visible in the accounts page, and same with the change username.
$ diff -y requirements_py2.txt requirements_py3.txt
setuptools==36.5.0 setuptools==36.5.0
cwl-runner cwl-runner
cwltool==1.0.20180403145700 | cwltool==1.0.20170828135420
schema-salad==2.7.20180514132321 | schema-salad==2.6.20170806163416
avro==1.8.1 | avro-cwl==1.8.3
ruamel.yaml==0.14.12 ruamel.yaml==0.14.12
requests==2.18.4 requests==2.18.4
Also we will create an issue for testing with python 3
Options for recording the Docker container identifier into a file:
--record-container-id
If enabled, store the Docker container ID into a file.
See --cidfile-dir to specify the directory.
--cidfile-dir CIDFILE_DIR
Directory for storing the Docker container ID file.
The default is the current directory
--cidfile-prefix CIDFILE_PREFIX
Specify a prefix to the container ID filename. Final
file name will be followed by a timestamp. The default
is no prefix.
Sorry @denis-yuen , we want to be able to point some monitoring tools to the correct containers via the cidfiles
@keiranmraine you can use cwltool directly, it speaks GA4GH TRS to dockstore.org https://github.com/common-workflow-language/cwltool#use-with-ga4gh-tool-registry-api
(I've raised a small, unrelated, Q on cwl gitter channel too)
@keiranmraine thanks, I've logged ga4gh/dockstore#1915 for us to update and test with a version of cwltool that supports the monitoring options
I'm executing a command:
dockstore tool launch --local-entry Dockstore.cwl --json test_run_L000013.json and as far as I can tell the container itself is happy. The pipeline within the container is throwing errors which I think are being saved to logs on the container-local filesystem. Can I view these logs? Does the container instance die when the pipeline terminates?
If you look here it seems like there are options to not delete the container after the pipeline is run
If you add these parameters to your config file then I think you should be able to view the logs by entering the container
If anything is unclear please let me know
At least, I'm able to get into it while it's running but as soon as it errors I get disconnected
It did give me a more detailed trackback though, so I guess that's good
Obviously I don't expect anyone to know specifics about the container I'm running, but does the error message
cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['WGS_*_vs_*.result.tar.gz']' bring to mind any general ideas what could be happening?
The command that I think is being run right before the error is
battenberg.pl -o /var/spool/cwl/WGS_2016-2066_vs_2016-2067/battenberg -u /var/spool/cwl/reference_files/battenberg/1000genomesloci -e /var/spool/cwl/reference_files/battenbe
rg/impute/impute_info.txt -c /var/spool/cwl/reference_files/battenberg/probloci.txt -r /var/spool/cwl/reference_files/genome.fa.fai -ig /var/spool/cwl/reference_files/battenberg/ignore_contigs.txt
-ge XX -tb /var/spool/cwl/tmp/2016-2066.bam -nb /var/spool/cwl/tmp/2016-2067.bam -p allelecount -nl 50 -t 40 >& /var/spool/cwl/timings/WGS_2016-2066_vs_2016-2067.time.alleleCount where I don't see anything like the glob above
The glob pattern is from your CWL tool description itself. Can you share your CWL files?
Dockstore has command examples for both
cwl-runner and cwltool but I find the cwltool versions fail to resolve linked files:$ cwltool --make-template quay.io/wtsicgp/dockstore-cgpmap:3.0.0
/usr/bin/cwltool 1.0.20180302231433
Resolved 'quay.io/wtsicgp/dockstore-cgpmap:3.0.0' to 'https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor'
No handlers could be found for logger "rdflib.term"
Tool definition failed initialization:
https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor:21:5: ('https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError(u'404 Client Error: Not Found for url: https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))
$ echo $?
1
vs
$ cwl-runner --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%2FDockstore.cwl
/usr/bin/cwl-runner 1.0.20180302231433
No handlers could be found for logger "rdflib.term"
bams_in: null
bwa: null
bwa_idx:
class: File
path: default/file/path
groupinfo: null
mmqc: false
mmqcfrac: null
reference:
class: File
path: default/file/path
sample: default_string
Also, that version of cwltool is 9 months out of date..