During my investigations on the reference quality of Wikidata, I extracted 6 subsets from Wikidata corresponding to 6 Wikidata WikiProjects. The outcomes were used for a paper that was submitted to Wikidata Workshop 2021. The GitHub repo for the paper contains ShEx schemas of each subset for the community reuse if you want to slurp the subsets by ShEx validators.
During my investigations on the reference quality of Wikidata, I extracted 6 subsets from Wikidata corresponding to 6 Wikidata WikiProjects. The outcomes were used for a paper that was submitted to Wikidata Workshop 2021. The GitHub repo for the paper contains ShEx schemas of each subset for the community reuse if you want to slurp the subsets by ShEx validators.
The extracted subsets (.nt.gz via WDumper) are in https://doi.org/10.5281/zenodo.5117927 BTW.
@metasj:matrix.org About subsetting at scale: First of all, just shex-js and PyShex support slurping. If you try to validate+slurp a schema in both over Wikdiata endpoint (without LIMIT) both validators will stop (PyShex will stop and shex-js face with the 'Maximum call stack size exceeded' error). There is no option to validate+slurp data from a local RDF file (instead of an endpoint).
About the reference slurping: Assume you want the references as complete as possible. If you want to extract references in a shape, you need to know the current properties that are used in the item. If another property will be added in the future, you need to update your shape with the newly added property. For example, assume all gene instaces have only P31. You can slurp references using this shape:<gene> {
p:P31 {
ps:P31 wd:Q7187 ;
prov:wasDerivedFrom {
pr:P813 xsd:dateTime ;
pr:P686 xsd:string ;
prv:P813 IRI ;
... } } ;}
If in the future, a P279 will be added to an instance of genes in Wikidata, you need to write a similar shape for it too. plus that all references don't have similar triples (some have just stated in. some have two, etc.).
I hope I could explain the problem here!
1 reply
p:<property>) work or not? Thanks a lot!
p:P279) is captured in statements which have triples with the ps form of the property (e.g. ps:P279) and are connected to provenance by a prov:wasDerivedFrom.
validate gives a good idea of the requires