These are chat archives for frictionlessdata/chat

2nd
May 2018
Adam Shepherd
@ashepherd
May 02 2018 13:33

Hi folks, I'm a huge fan of the Frictionlessdata spec, and we've started to roll this out for our NSF-funded data repository for bio & chem oceanography data. We provide data managment support for individual researchers on short-term funding (1-3 yrs). I ran into a potential issue with the 'missingValues' being at the table-level where the pre-v1 spec had a similar descriptor at the field level.

Our researcher's data contain short-hand values that can be joined with lookup tables after they come back from sea. These fields are usually things like species names or quality flags that they abbreviate to save time during collection. I haven't processed all our data to detect if we have a collision between a missingValues string meant for one column and a legitimate data value in another column, but there could be a scenario where a missing data value of 'nd' in one column might be a valid data value in another column. I saw that it got mentioned here https://github.com/frictionlessdata/specs/issues/429#issuecomment-304601058 but was curious if the idea of supporting field-level missingValues was a possibility? cheers, and thank you all!

Vitor Baptista
@vitorbaptista
May 02 2018 14:15
Hi @ashepherd, I faced a similar issue a few months ago and we started discussing it on frictionlessdata/specs#551. We agreed on allowing missingValue field both on the table and field levels (in cases where both are present, the more specific wins). We just haven't got around doing it yet.
It would be good if you could add your use case in that issue as well.
(BTW, great to know about your usage in the data repository. Is it publicly accessible? If so, could you share its URL?)
@Stephen-Gates already started writing the spec in https://discuss.okfn.org/t/missing-values-per-field-pattern/6571
Rufus Pollock
@rufuspollock
May 02 2018 14:20

Hi @ashepherd, I faced a similar issue a few months ago and we started discussing it on frictionlessdata/specs#551. We agreed on allowing missingValue field both on the table and field levels (in cases where both are present, the more specific wins). We just haven't got around doing it yet.

:thumbsup: :clap: