tableschema and then datapackage It tells me datapackage -> cchardet[version='>=1.0,<2.0'] -> python=3.6 but in the setup.py for datapackage I can see that the current requirement is only >=1.0 and that that was updated some months ago.
datapackage-py repo. There's a mismatch between the requirements that show up in conda-forge and those enumerated in setup.py for v1.6.0 Not sure how that happened.
Hi! Is a tabular data package with both path and data valid and acceptable to tools where the data and CSV file are numerically equivalent?
generally not, you need one. of them. BTW there is a convention that if you want to "cache" the value of a path file onto the Data Resource (a common practice e.g. for visualization) you store in _values attribute see e.g. https://frictionlessdata.io/specs/views/
I've been doing some thinking about the parallels between a Data Package-oriented solution design compared to OpenAPI-driven development, and it seems to me to be a valid use case. If there's any interest in having a lightweight standalone tool, let me know.
This - https://github.com/frictionlessdata/tableschema-pandas-py - suggests I should be able to pull an existing pandas dataframe into a data package:
>>> datapackage.pull_datapackage('/tmp/datapackage.json', 'country_list', 'pandas', tables={
... 'data': storage['data___data'],
... })But apparently pull_datapackage is deprecated. Can someone suggest what the updated version of this should be?
package = Package(storage=storage)) and save it (push) using the saving method (package.save(storage=storage))
@roll
502 error on https://goodtables.io ? Still happens to me >50% of the time and seems related to the cloudflare setup.
I've been gone for a while, was wanting to get back into goodtables.io. Unfortunately the Cloudflare 502 error is still happening.
A module for mapping to/from data packages to sqlalchemy would be super useful.
Could be used for getting to graphql also (@loleg): https://github.com/graphql-python/graphene-sqlalchemy
Some prior work in this gist: https://gist.github.com/LuizArmesto/025aeba8f5c6d6f058ee
tableschema-sql driver? It's not fully documented yet but it can be used with datapackage the same way as with tableschema - https://github.com/frictionlessdata/tableschema-py/blob/master/examples/table_sql.py
from sqlalchemy import create_engine
from datapackage import Package, Resource
DATABASE_URL = 'postgresql://roll:roll@localhost:5432/movies'
# Option 1
descriptor = {'resources': [{'path': 'countries'}, {'path': 'locations'}]}
package = Package(descriptor, storage='sql', engine=create_engine(DATABASE_URL))
package.infer()
package.save('descriptor.json')In the data resource spec, there are two inline data examples:
{
"data": {
"resource-name-data": [
{"a": 1, "b": 2}
]
},
}and
{
"resources": [
{
"format": "json",
# some json data e.g.
"data": [
{ "a": 1, "b": 2 }
]
}
]
}Are they both correct? is the idea of the nested resource-name-data option to be analogous to "path": [ "myfile1.csv", "myfile2.csv" ]?
{
"data": {
"resource-name-data": [
{"a": 1, "b": 2}
]
},
}tabular data resource the data field must be an array - https://frictionlessdata.io/specs/tabular-data-resource/
@loleg
Let's brainstorm the scope for a joint project ..
@loleg I'm currently playing around with tableschema-backed attrs class generation and a corresponding mypy plugin. The plugin is difficult. I'm interested in what it would take to make a strongly typed, performant data pipeline, possibly with apache beam compatibility
dialect.header=false - http://frictionlessdata.io/specs/csv-dialect/