Also we will create an issue for testing with python 3
Options for recording the Docker container identifier into a file:
--record-container-id
If enabled, store the Docker container ID into a file.
See --cidfile-dir to specify the directory.
--cidfile-dir CIDFILE_DIR
Directory for storing the Docker container ID file.
The default is the current directory
--cidfile-prefix CIDFILE_PREFIX
Specify a prefix to the container ID filename. Final
file name will be followed by a timestamp. The default
is no prefix.
Sorry @denis-yuen , we want to be able to point some monitoring tools to the correct containers via the cidfiles
@keiranmraine you can use cwltool directly, it speaks GA4GH TRS to dockstore.org https://github.com/common-workflow-language/cwltool#use-with-ga4gh-tool-registry-api
(I've raised a small, unrelated, Q on cwl gitter channel too)
@keiranmraine thanks, I've logged ga4gh/dockstore#1915 for us to update and test with a version of cwltool that supports the monitoring options
I'm executing a command:
dockstore tool launch --local-entry Dockstore.cwl --json test_run_L000013.json and as far as I can tell the container itself is happy. The pipeline within the container is throwing errors which I think are being saved to logs on the container-local filesystem. Can I view these logs? Does the container instance die when the pipeline terminates?
If you look here it seems like there are options to not delete the container after the pipeline is run
If you add these parameters to your config file then I think you should be able to view the logs by entering the container
If anything is unclear please let me know
At least, I'm able to get into it while it's running but as soon as it errors I get disconnected
It did give me a more detailed trackback though, so I guess that's good
Obviously I don't expect anyone to know specifics about the container I'm running, but does the error message
cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['WGS_*_vs_*.result.tar.gz']' bring to mind any general ideas what could be happening?
The command that I think is being run right before the error is
battenberg.pl -o /var/spool/cwl/WGS_2016-2066_vs_2016-2067/battenberg -u /var/spool/cwl/reference_files/battenberg/1000genomesloci -e /var/spool/cwl/reference_files/battenbe
rg/impute/impute_info.txt -c /var/spool/cwl/reference_files/battenberg/probloci.txt -r /var/spool/cwl/reference_files/genome.fa.fai -ig /var/spool/cwl/reference_files/battenberg/ignore_contigs.txt
-ge XX -tb /var/spool/cwl/tmp/2016-2066.bam -nb /var/spool/cwl/tmp/2016-2067.bam -p allelecount -nl 50 -t 40 >& /var/spool/cwl/timings/WGS_2016-2066_vs_2016-2067.time.alleleCount where I don't see anything like the glob above
The glob pattern is from your CWL tool description itself. Can you share your CWL files?
Dockstore has command examples for both
cwl-runner and cwltool but I find the cwltool versions fail to resolve linked files:$ cwltool --make-template quay.io/wtsicgp/dockstore-cgpmap:3.0.0
/usr/bin/cwltool 1.0.20180302231433
Resolved 'quay.io/wtsicgp/dockstore-cgpmap:3.0.0' to 'https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor'
No handlers could be found for logger "rdflib.term"
Tool definition failed initialization:
https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor:21:5: ('https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError(u'404 Client Error: Not Found for url: https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))
$ echo $?
1
vs
$ cwl-runner --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%2FDockstore.cwl
/usr/bin/cwl-runner 1.0.20180302231433
No handlers could be found for logger "rdflib.term"
bams_in: null
bwa: null
bwa_idx:
class: File
path: default/file/path
groupinfo: null
mmqc: false
mmqcfrac: null
reference:
class: File
path: default/file/path
sample: default_string
Also, that version of cwltool is 9 months out of date..
and when it works the output is much nicer:
$ cwltool --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%Fwtsicgp%Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%FDockstore.cwl
/home/mcrusoe/cwltool/env3/bin/cwltool 1.0.20181217162649
sample: a_string # type "string"
reference: # type "File"
class: File
path: a/file/path
mmqcfrac: 0.05 # default value of type "float". (optional)
mmqc: false # type "boolean"
groupinfo: # type "File" (optional)
class: File
path: a/file/path
bwa_idx: # type "File"
class: File
path: a/file/path
bwa: ' -Y -K 100000000' # default value of type "string". (optional)
bams_in: # array of type "File" (optional)
- class: File
path: a/file/path
An interesting issue here:
I have a workflow registerd on Dockstore, it runs fine with this command:
dockstore workflow launch --entry github.com/cancerit/workflow-seq-import/chksum_seqval_wf_interleaved_fq:0.4.0 --json chksum_seqval_wf_interleaved_fq_with_headers.jsonit'll fail if I run it like below:
cwl-runner https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/%2Fcwls%2Fchksum_seqval_wf_interleaved_fq.cwl chksum_seqval_wf_interleaved_fq_with_headers.json
Tool definition failed initialization:
Tool definition https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl failed validation:
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:27:1: checking field `steps`
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:28:3: checking object `https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl#if_file_name_is_bz2_then_return_null_else_return_in_json_to_output`
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:35:5: Field `run` contains undefined reference to `https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl`
too old?
Not Found : version found, but file not found (bad filename, invalid file, etc.)
Looks like an issue with dockstore.org itself
One other thought curl -X GET "https://dockstore.org/api/api/ga4gh/v2/tools/%2523workflow%252Fgithub.com%252Fcancerit%252Fworkflow-seq-import%252Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/CWL/files" -H "accept: application/json" gives a list of files that we're aware of
https://dockstore.org/api/api/ga4gh/v2/tools/%2523workflow%252Fgithub.com%252Fcancerit%252Fworkflow-seq-import%252Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/PLAIN_CWL/descriptor/if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl
is listed but the CWL (output) seems to refer to a toolkit directory
is listed but the CWL (output) seems to refer to a toolkit directory
Skipping dockstore.org, everything seems to validate okay: https://view.commonwl.org/workflows/github.com/cancerit/workflow-seq-import/blob/0.4.0/cwls/chksum_seqval_wf_interleaved_fq.cwl
l$ cwltool --validate https://github.com/cancerit/workflow-seq-import/raw/0.4.0/cwls/chksum_seqval_wf_interleaved_fq.cwl
/home/michael/cwltool/env3/bin/cwltool 1.0.20181217162649
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
https://github.com/cancerit/workflow-seq-import/raw/0.4.0/cwls/chksum_seqval_wf_interleaved_fq.cwl is valid CWL.
Yeah, I think it looks like an indexing error. For some reason, it looks like toolkit/if_input_is_bz2_convert_to_gz_else_just_rename.cwl and toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl are being indexed in the toolkit directory. if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl is being indexed in the pwd, possibly because its an import in an import. What I can't explain is why it launches properly using the very first command but I think its still a bug one way or another. I'll create an issue for it