Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Dominic Fitzgerald
@djf604
Obviously I don't expect anyone to know specifics about the container I'm running, but does the error message cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['WGS_*_vs_*.result.tar.gz']' bring to mind any general ideas what could be happening?
The command that I think is being run right before the error is battenberg.pl -o /var/spool/cwl/WGS_2016-2066_vs_2016-2067/battenberg -u /var/spool/cwl/reference_files/battenberg/1000genomesloci -e /var/spool/cwl/reference_files/battenbe rg/impute/impute_info.txt -c /var/spool/cwl/reference_files/battenberg/probloci.txt -r /var/spool/cwl/reference_files/genome.fa.fai -ig /var/spool/cwl/reference_files/battenberg/ignore_contigs.txt -ge XX -tb /var/spool/cwl/tmp/2016-2066.bam -nb /var/spool/cwl/tmp/2016-2067.bam -p allelecount -nl 50 -t 40 >& /var/spool/cwl/timings/WGS_2016-2066_vs_2016-2067.time.alleleCount where I don't see anything like the glob above
Michael R. Crusoe
@mr-c
@djf604 please re-run directly with the latest cwltool release. There should be no /var/spool/cwl paths used
The glob pattern is from your CWL tool description itself. Can you share your CWL files?
Keiran Raine
@keiranmraine
Dockstore has command examples for both cwl-runner and cwltool but I find the cwltool versions fail to resolve linked files:
$ cwltool --make-template quay.io/wtsicgp/dockstore-cgpmap:3.0.0
/usr/bin/cwltool 1.0.20180302231433
Resolved 'quay.io/wtsicgp/dockstore-cgpmap:3.0.0' to 'https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor'
No handlers could be found for logger "rdflib.term"
Tool definition failed initialization:
https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor:21:5: ('https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError(u'404 Client Error: Not Found for url: https://dockstore.org:8443/api/ga4gh/v1/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))
$ echo $?
1
vs
$ cwl-runner --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%2FDockstore.cwl
/usr/bin/cwl-runner 1.0.20180302231433
No handlers could be found for logger "rdflib.term"
bams_in: null
bwa: null
bwa_idx:
    class: File
    path: default/file/path
groupinfo: null
mmqc: false
mmqcfrac: null
reference:
    class: File
    path: default/file/path
sample: default_string
Michael R. Crusoe
@mr-c
@keiranmraine Can you file an issue on the cwltool repo?
Also, that version of cwltool is 9 months out of date..
Keiran Raine
@keiranmraine
Just verifying with latest version. At first wasn't sure if it was a doc issue on Dockstore as workflows don't use the quay.io url... I don't think Dockstore indicates support for the latest version of cwltool
Michael R. Crusoe
@mr-c
I can reproduce the problem with latest cwltool
and when it works the output is much nicer:
$ cwltool --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%Fwtsicgp%Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%FDockstore.cwl
/home/mcrusoe/cwltool/env3/bin/cwltool 1.0.20181217162649
sample: a_string  # type "string"
reference:  # type "File"
    class: File
    path: a/file/path
mmqcfrac: 0.05  # default value of type "float". (optional)
mmqc: false  # type "boolean"
groupinfo:  # type "File" (optional)
    class: File
    path: a/file/path
bwa_idx:  # type "File"
    class: File
    path: a/file/path
bwa: ' -Y -K 100000000'  # default value of type "string". (optional)
bams_in:  # array of type "File" (optional)
  - class: File
    path: a/file/path
Keiran Raine
@keiranmraine
Nice! Will raise it now
Keiran Raine
@keiranmraine
Yaobo Xu
@byb121

An interesting issue here:
I have a workflow registerd on Dockstore, it runs fine with this command:

dockstore workflow launch --entry github.com/cancerit/workflow-seq-import/chksum_seqval_wf_interleaved_fq:0.4.0 --json  chksum_seqval_wf_interleaved_fq_with_headers.json

it'll fail if I run it like below:

cwl-runner https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/%2Fcwls%2Fchksum_seqval_wf_interleaved_fq.cwl chksum_seqval_wf_interleaved_fq_with_headers.json
Tool definition failed initialization:
Tool definition https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl failed validation:
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:27:1:   checking field `steps`
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:28:3:     checking object `https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl#if_file_name_is_bz2_then_return_null_else_return_in_json_to_output`
https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl:35:5:       Field `run` contains undefined reference to `https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fcancerit%2Fworkflow-seq-import%2Fchksum_seqval_wf_interleaved_fq/versions/0.4.0/plain-CWL/descriptor/toolkit/if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl`
Michael R. Crusoe
@mr-c
@byb121 cwl-runner --version ?
Yaobo Xu
@byb121
/usr/local/bin/cwl-runner 1.0.20180403145700
too old?
Michael R. Crusoe
@mr-c
Probably :-)
Yaobo Xu
@byb121
sure, I'll have a go again on the latest version.
Looks like an issue with dockstore.org itself
Denis Yuen
@denis-yuen
Yaobo Xu
@byb121
latest version of cwl-runner returns the same error.
Michael R. Crusoe
@mr-c
l$ cwltool --validate https://github.com/cancerit/workflow-seq-import/raw/0.4.0/cwls/chksum_seqval_wf_interleaved_fq.cwl
/home/michael/cwltool/env3/bin/cwltool 1.0.20181217162649
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:name' not recognized, are you missing a $namespaces section?
URI prefix 'foaf' of 'foaf:mbox' not recognized, are you missing a $namespaces section?
https://github.com/cancerit/workflow-seq-import/raw/0.4.0/cwls/chksum_seqval_wf_interleaved_fq.cwl is valid CWL.
Yaobo Xu
@byb121
if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl is indeed in a directory called toolkit
Denis Yuen
@denis-yuen
Yeah, I think it looks like an indexing error. For some reason, it looks like toolkit/if_input_is_bz2_convert_to_gz_else_just_rename.cwl and toolkit/if_input_is_bz2_generate_md5sum_else_return_input_chksum_json.cwl are being indexed in the toolkit directory. if_file_name_is_bz2_then_return_null_else_return_in_json_to_output.cwl is being indexed in the pwd, possibly because its an import in an import. What I can't explain is why it launches properly using the very first command but I think its still a bug one way or another. I'll create an issue for it
Yaobo Xu
@byb121
Thanks.
When it's fixed, will this registry be fixed automatically?
Denis Yuen
@denis-yuen
Created ga4gh/dockstore#2154 , so chances are if its an indexing error. We'll need to fix the refresh behaviour and then you'll need to hit the refresh button when we do a release
I'm having difficulty explaining why the first launch command works since it uses cwltool under the hood, so there may be something more complicated I'm missing. But that's my first explaination
Yaobo Xu
@byb121
That's why it's interesting :)
Yasset Perez-Riverol
@ypriverol
Hi everyone, Im happy to announce that Biocontainers has implemented already the TRS implementation for tool and tool versions
We are working towards to have all the endpoints
Peter Amstutz
@tetron
@ypriverol oh cool
Yasset Perez-Riverol
@ypriverol
Thanks @denis-yuen for your support
We have done some changes to the API and probably a PR will come next week
Denis Yuen
@denis-yuen
Ah, cool. Just in time for the Monday GA4GH TRS update!
Michael R. Crusoe
@mr-c
@ypriverol Yay!
Yasset Perez-Riverol
@ypriverol
for example, Apart of docker containers must of our consumers use singularity or "Conda containers"
we have extend the classification to container image to provide more details about the tipe of container
like conda package, or singularity or Docker
this will allow us to connect to workflows that use conda for example
make sense?
Peter Amstutz
@tetron
does that mean there are/will be CWL tool wrappers for biocontainers?
Yasset Perez-Riverol
@ypriverol
I hope
Denis Yuen
@denis-yuen
@ypriverol I think that makes sense, we'll see/discuss more in the PR I think