Grunt is not needed
📰 Want to track progress on the big FD v2 update? See this issue …
We’re making great progress in our sprint this week and here’s the current high level status
@augusto-herrmann correct and fixed. 417 is the focus - though we are nearing completion. Maybe best item would be to look at frictionlessdata/forum#5 and see if you can liase with @johanricher on a plan of work there ...
@lwinfree (and others), hi! Great talk at https://fosdem.org/2020/schedule/event/open_research_frictionless_data/ ! I believe you mentioned an integration with Zenodo but I'm having trouble finding any information about this. Can someone please drop a link in here?
Hi @pdurbin! For Zenodo, I was refering to our recent Pilot collaboration with Catalyst Co-op, where they packaged US energy data into datapackages & then uploaded to Zenodo. I think this issue is the best place to read a summary: catalyst-cooperative/pudl#425 See also: https://github.com/catalyst-cooperative/pudl/blob/master/src/pudl/load/metadata.py & https://github.com/catalyst-cooperative/pudl/blob/master/docs/datapackages.rst for more info. Let me know if you have other questions, or if you’d like to talk to the Catalyst team!
Hey, so it's been a long time since I've checked in. I wrote a partial Swift language implementation of [Tabular] Data Package and Table Schema for my own project (which is used in the shipped Quotemarks iOS app). It is unlikely I'll have the time/energy to complete the entire specification (e.g. zipping, data inference, every single data type). I haven't yet open sourced it since it's incomplete. If someone on here wants to complete the Swift work, reach out to me. Implementation status is here: frictionlessdata/software#29
I did write a dialectal CSV parser in Swift since one did not exist already. That's open sourced here: https://github.com/slythfox/csv-dialect-swift
@slythfox It would be great to have it in the implementations family. I think at some point we will find how to finish it cc @lwinfree
Frictionless Data keeps coming up in my world. Just now: https://github.com/IQSS/dataverse/issues/6678#issuecomment-591564656
Frictionless Data keeps coming up in my world. Just now: https://github.com/IQSS/dataverse/issues/6678#issuecomment-591564656
😄is there anything we can contribute to that discussion?
@roll @rufuspollock I opened an issue a month ago about multipart header frictionlessdata/datapackage-py#256
I have a commit ready for review in there, should I create a PR about this ?
I have a commit ready for review in there, should I create a PR about this ?
Hi all, I couldn't find in the docs a way to get a report generated by goodtables.validate(..) including the actual row data (tables.errors.row), I can only retrieve the row-number (tables.errors.row-number). Am I missing some config/flag to enable this behaviour? http://try.goodtables.io/ is showing the actual cell values. Many thanks and great job!
@didiez thanks for asking - could you open an issue in the forum with details and we’ll try to debug / answer there https://github.com/frictionlessdata/forum/issues
Our schema allows extra headers and missing non-required headers, so the csv could have less (or more) column than the defined schema.
Traceback (most recent call last):
File "test.py", line 8, in <module>
report = validate("datapackage.json", checks=["structure", "schema", "foreign-key"], order_fields=True, infer_fields=False)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/goodtables/validate.py", line 80, in validate
report = inspector.inspect(source, **options)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/goodtables/inspector.py", line 82, in inspect
table_warnings, table_report = task.get()
File "/home/didiez/anaconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
File "/home/didiez/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/didiez/anaconda3/lib/python3.7/site-packages/goodtables/inspector.py", line 200, in __inspect_table
success = prepare_func(stream, schema, extra)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/goodtables/contrib/checks/foreign_key.py", line 48, in prepare
current_resource_name=extra['resource-name'])
File "/home/didiez/anaconda3/lib/python3.7/site-packages/goodtables/contrib/checks/foreign_key.py", line 116, in _get_relations
relations[resource_name] = resource.read(keyed=True)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/datapackage/resource.py", line 377, in read
foreign_keys_values=foreign_keys_values, **options)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/tableschema/table.py", line 353, in read
for count, row in enumerate(rows, start=1):
File "/home/didiez/anaconda3/lib/python3.7/site-packages/tableschema/table.py", line 215, in iter
for row_number, headers, row in iterator:
File "/home/didiez/anaconda3/lib/python3.7/site-packages/tableschema/table.py", line 509, in builtin_processor
row, row_number=row_number, exc_handler=exc_handler)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/tableschema/schema.py", line 266, in cast_row
error_data=keyed_row)
File "/home/didiez/anaconda3/lib/python3.7/site-packages/tableschema/helpers.py", line 90, in default_exc_handler
raise exc
datapackage.exceptions.CastError: Row length 5 doesn't match fields count 6 for row "2"If we only check 'structure' and 'schema' everything works as expected.
I could not find any reference in the docs about reading (and casting) csv files with fields not defined in the schema o missing fields (non-required column in schema).
Should I open an issue or it's a known limitation when reading csv files with goodtables?
^ in case anyone here knows Workbench, please feel free to comment
@roll issue added frictionlessdata/goodtables-py#347. As a workaround I ended up pre-processing the source csv with pandas to add/remove/reorder columns to make it valid against the schema
Hi everyone, I'm happy to share that I recently joined the Frictionless Data team as a Developer Evangelist. My role involves spreading the word about Frictionless Data and encouraging community involvement. I'm always open to help and have discussions about Frictionless Data. https://www.datopian.com/blog/2020/03/20/joining-the-frictionless-data-team/
the error
No such file or directory: 'inline' was masking the actual error. To read from a file stream it must be binary, you must specify the format and it cannot live in a tmp path. Unfortunately the logic behind _local_file_not_found inside inspector.py - return urlparse(source).scheme == '' and not os.path.isfile(source) - masks the true error.
Hi, @roll I'm having issues getting goodtables js library to work. I have a demo here https://repl.it/@lauragift21/goodtables-js and this is the error returned when trying to validate a CSV file.
(node:66) UnhandledPromiseRejectionWarning: Error: Can't create a job on API. Reason: "Error: Request failed with status code 403"
@lauragift21 Please create an issue I'll investigate.
goodtables-js is a goodtables.io wrapper so something wrong with the API endpoint
I'm trying to figure out a solution to validate the range of dates in a CSV. Unfortunately the Excel short date format varies by locale, ie: dd/mm/yyyy in most of the world and dd/mm/yyyy in the US and US-centric places like Abu Dhabi. I might have to do multiple validation passes - the first to see if the it's a well-formed date, second to peak and auto detect the format, and 3rd to validate the range i.e. [dd/mm/yyyy, dd/mm/yyyy] or [mm/dd/yyyy, mm/dd/yyyy]. Has anyone attempted to solve a similar problem?