@roll OK, so it sounds like you are of the opinion that missing values should be ignored in uniqueness constraints. (I don't agree that this is clear from the specs, since "value" in "all values for that field MUST be unique" could be understood to include
null values). tableschema-py does what you say it should. However, goodtables-py does the opposite: 2 or more null values in a field triggers a unique-constraint error.
Hi guys, I believe the JSON schema files for data-package.json and data-resource.json on the frictionlessdata site are wrong and contradict the specs in respect on how a schema for a data resource can be given. I opened a ticket (frictionlessdata/specs#645) and would ask you to check whether I am am right here or seeing things.
Hi here,
I am trying to use goodtable in python to validate a single row and struggling to get it working with the internals. A bit of background I am using the validation part of goodtables to identify where my data is malformed. I then run a set of functions against the errored row in an attempt to fix it. Ideally I would then like to do something like
Thanks
I am trying to use goodtable in python to validate a single row and struggling to get it working with the internals. A bit of background I am using the validation part of goodtables to identify where my data is malformed. I then run a set of functions against the errored row in an attempt to fix it. Ideally I would then like to do something like
validate_row(row, schema) so that I can see if the fix worked or not. Atm it seems like the simplest solution is for me is to basically create a new descriptor that has the single row as the data but that seems inefficient to me? I just wanted to check that I hadn't missed anything or if there is an easier or more efficient way to validate a single row?Thanks
Although, it's not a really performant solution. I would rather use
tableschema.Schema.cast_row - https://github.com/frictionlessdata/tableschema-py#schemacast_rowrow
it
I think it's useful for datapackage with large volume
Dear all, I am organising a thematic track at FOSDEM (Free Open Source Software Developer European Meeting) conference this year called "Open Research Tools and Technologies". Having the frictionless data tools presented there would be neat ! Feel free (and actually encouraged) to apply to our call for presentations. Deadline 1rst December 2019. Conf at Bruxelles on Sturday the 1st February 2020. Details here: https://research-fosdem.github.io/
Ping me if you have any questions.
That's great! cc @lwinfree
Hi. I am working on a proof of concept using Frictionless data for capturing/sharing complex data for risk modelling. I had a few general questions, and appreciative of any assistance:
- Is it possible to reuse table schemas between data packages, perhaps by including them from an external file?
- Is it possible to reuse tables across data packages? We have rich domain data that we need to validate the data.
- Is it possible to define compound foreign key constraints?
hi @mpinkerton-oasis , @roll will confirm these, but my understanding is that the answer is yes to all:
- You can use a URL for the
schemaproperty of resources, pointing to a JSON Table Schema file hosted elsewhere - Do you mean different data pacakages having resources that point to the same data file? If so, I don't see why not as the
pathproperty can also be a URL - According to the spec (https://frictionlessdata.io/specs/table-schema/#foreign-keys) you can define a list of
fieldsboth in the source and the destination of the foreign key so that should cover compound FKs, unless I missunderstood
Thanks @amercader, on my second point the use case is having one data package that has a collection of domain data that is then used for key validation in a range of other data packages. I'm not sure if there is an established pattern for this, but if the schema and data can both be set as paths that will certainly help.
@mpinkerton-oasis
We have recently added an external FKs support for goodtables:
- https://github.com/frictionlessdata/goodtables-py#foreign-key
- https://github.com/frictionlessdata/goodtables-py/blob/master/tests/contrib/checks/test_foreign_key.py#L118-L144
The second example shows how it uses the country-codes data package as a reference table.
well actually my current issue could also be solved by changing the datapackage-js behaviour. Currently the issue is that with a package which has a group of resources, if a schema is indicated for all the resources of the group (which are numerous in my case > 1000) then the lib will load the same schema as many times as the number or resources
which is not ideal... above all if the basepath is actually remote...
Thus I see two ways out :
- a change my package by removing schema in my grouped resources but the first one
- I update the datapackage-js to load only the first schema is resources are in a group
Any comments ?
@paulgirard this is a great question - i think it will require a bit of thought - can you post this in our new trial forum for technical questions in github issues https://github.com/frictionlessdata/forum/issues
Note : I'll try the first approach as a quick workaround but I can do the dev for the second path if validated
ah ah I just wrote the first issue ;-)
By the way @lwinfree, very happy to have seen your proposal in our devroom at the FOSDEM (see https://research-fosdem.github.io/). We finished reviewing this morning : you are in !
Glad to meet you and share frictionless data with FOSDEM audience in Brussels next year :)
@paulgirard i’ll comment there too - one point reading your question is that you have a use case for chunks. There seem several solutions …
- Use chunks (but that is not yet properly supported in the libs)
- Cache the remote url so you don’t do a network call a 1000x
- … maybe others ...
yes caching would do it. We could add a caching mechanism into the datapackage-js which don't load twice the same schema file.
which could be maybe be done easily with https://github.com/jin5354/axios-cache-plugin