Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 07 21:16
    jonrmitchell opened #759
  • Dec 07 16:14
    sapetti9 edited #703
  • Dec 07 16:14
    sapetti9 edited #703
  • Dec 07 16:14
    sapetti9 edited #703
  • Dec 07 16:13
    sapetti9 edited #703
  • Dec 06 11:02

    github-actions[bot] on gh-pages

    Deploying to gh-pages from @ 3… (compare)

  • Dec 06 10:59

    roll on master

    Removed double dot from a descr… (compare)

  • Dec 06 08:23

    github-actions[bot] on gh-pages

    Deploying to gh-pages from @ 0… (compare)

  • Dec 06 08:19
    roll closed #756
  • Dec 06 08:19

    roll on master

    Schema/dialect should accept st… (compare)

  • Dec 06 07:24

    github-actions[bot] on gh-pages

    Deploying to gh-pages from @ e… (compare)

  • Dec 06 07:20

    roll on master

    Fixed deploy (compare)

  • Dec 06 06:59

    roll on master

    Add name data resource to the d… (compare)

  • Dec 06 06:59
    roll closed #757
  • Dec 06 05:13
    AFoletti opened #758
  • Nov 29 17:29
    gabrielbdornas opened #757
  • Nov 26 09:36
    sapetti9 edited #703
  • Nov 26 09:35
    sapetti9 closed #701
  • Nov 26 09:35
    sapetti9 edited #703
  • Nov 26 09:35
    sapetti9 opened #703
Rufus Pollock
@rufuspollock
@jungshadow we'll probably be able to look at this properly early in the new year (i.e. late next week). hope that works :-)
Rufus Pollock
@rufuspollock
@vitorbaptista how are you doing with your review of the data lib APIs? Worth a checkin?
Rufus Pollock
@rufuspollock

Interested in a good tables in javascript?

I'd like to know if there is anyone else interested in a goodtables style library in pure javascript.

:thumbsup: if you are interested.

Note: We already have the start of this in the tableschema-js here and the work would involve splitting this out and making its reporting compatible with current goodtables python implementation.

Eoghan Ó Carragáin
@eocarragain
Hi @rufuspollock @danfowler. I'm putting in a proposal for a Research Data Alliance bird of a feather session to look at the various approaches to packaging research data. Here is a list of packaging specifications I've identified which obviously includes frictionless data: https://docs.google.com/document/d/155lA2BcixTl-zwJHGfLkxsmg7WmQbBK00QWyP8QggkE/edit
The deadline for the RDA BoF is today. Here is my draft proposal. Sorry for not getting in touch about this earlier & I don't expect anyone to read it today! However, if the session is accepted, it would be great to have somebody present or speak to the frictionless data specs. I'm also hoping somebody from the ResarchObjects projects and others may be available
cheers
Rufus Pollock
@rufuspollock
@eocarragain that sounds great!
Eoghan Ó Carragáin
@eocarragain
@rufuspollock great! will let you know how the submission goes.
Jared
@jungshadow

@jungshadow we'll probably be able to look at this properly early in the new year (i.e. late next week). hope that works :-)

Perfect, @rufuspollock! Apologies if I was being overbearing about the issue. I wasn’t sure when or how issues are queued for assessment and may have let my unfounded expectations get the best of me :) Again, appreciate all @roll’s help and appreciate the consideration on the potential modification. Happy holidays and all the best in the new year!

Rufus Pollock
@rufuspollock
@jungshadow not all -- thank-you for prompting :-)
Stephen Gates
@Stephen-Gates
@rufuspollock :+1: to goodtables.js. I can see plenty of uses for the library in Data Curator. Top ones being providing a data quality report within the Data Package and allowing advanced data validation checks.
Rufus Pollock
@rufuspollock

@roll @vitorbaptista do we have a common repo(s) where we have sample data files e.g. sample csv, sample data packages etc?

If not what do you think of maintaining one? It would be super useful for the community to have a common set of test data.

Stephen Gates
@Stephen-Gates
^^ agreed current example repos are out of date e.g. https://github.com/frictionlessdata/example-data-packages
Rufus Pollock
@rufuspollock
@Stephen-Gates :thumbsup:
@Stephen-Gates do you know of any other example datasets or files?
I also started making one for Data Curator testing but it’s a bit of a mess at present https://github.com/Stephen-Gates/data-package-examples
Rufus Pollock
@rufuspollock

Thinking about requirement for example data packages:

  1. As a Tutorial writer I want a set of data files and data packages I can use in my tutorials so that i can embed them and point users to them to play with them themselves
  2. As a Developer writing a library I want to have a set of standard test data files and data packages as a reference for my implementation tests
  3. As a new Publisher of data packages i want to see examples that i can copy and use so that I can move quickly and understand what is involved
  4. As a Consumer of data packages I want to see some examples for use

My sense is that the "exemplar" and "test" use cases are somewhat different. 1+3+4 are exemplar and want "nice' data packages". 2 (+1) are more test and are about testing the real range of sitautions and being super simple for testing.

My sense is that the key here to focus on is the test (lib developer) case to start with.

They probably want versioning and ability to git submodule so they can pin the data they are developing against (e.g. if data package spec gets upgraded they can still keep old spec versions if they need them).

wdyt?

Stephen Gates
@Stephen-Gates
@rufuspollock sounds good. Agree on differences between examplar and test. Some of my packages have data errors on purpose that have help discover issues in table schema.js and data curator.
Rufus Pollock
@rufuspollock
@Stephen-Gates exactly
@vitorbaptista @pwalsh where is the best place to open issues about this kind of thing? I think the pm repo has been deprecated / deleted https://github.com/frictionlessdata/pm 404s ...
roll
@roll
@rufuspollock I think Dan has started this work in https://github.com/frictionlessdata/example-data-packages

About issues current system is we have two main issue trackers aside concrete libs:

The second on I think could be used as just for everything else except the specs. Also the FD workboard lives on this repo:

roll
@roll
@rufuspollock @Stephen-Gates Have you eliminated the idea of having goodtables-py/CLI as a backend for goodtables-js - frictionlessdata/goodtables-js#19 It's a pretty trivial feature to implement which allow to validate data in JavaScript locally using Python CLI to generate data quality reports. Having pure goodtables-js is cool but we should take into account that it could mean really a lot of work because in JavaScript we miss both tabulator and goodtables. Which I would say the most complex software in the Python stack. Also it could be always a problem that JavaScript implementation will be possibly always very limited in compare to the Python one. Python for example already support advanced checks and other advanced concepts.
roll
@roll
Last but not least I wish everyone Happy Merry Christmas (retrospectively=) and Happy New Year! I would say this year was absolutely stellar for the whole Frictionless Data community. The movement have finished the specs-v1 and implemented it for 9(!!!) languages. There are now goodtables.io stack, new-brand datahub.io, awesome projects like Data Curator and much more. It's really great to be a part of this project and community. :tada: :sparkles: :+1:
nathanxmeyer
@nathanxmeyer
Hello frictionlessdata folks. New here (to python, to frictionlessdata, etc). I am working on a personal project in the archaeology domain, using python to build a SQLite database from many CSV files. My first question is whether the frictionless data schema specifications for datapackage and table are fairly stable. My second question is whether the goodtables python implementation is something that has future viability. Last question: is there a good example of validation against a JSON schema - the readme references this but I am not seeing in the documentation how to pass the schema to validate. Thanks very much, Nathan
nathanxmeyer
@nathanxmeyer
Hello again, I found this (https://frictionlessdata.io/guides/validating-data/) so am fine on my last question but still interested in thoughts on the first two. Thanks!
Rufus Pollock
@rufuspollock

@nathanxmeyer first, great to hear from you. To answer your questions:

My first question is whether the frictionless data schema specifications for datapackage and table are fairly stable.

Yes, the schema specs are definitely very stable - they are now v1.0 and have been refined for ~5y. Of course, they will continue to evolve gently but backwards compatibility will be maintained etc.

My second question is whether the goodtables python implementation is something that has future viability

Yes, absolutely. These have been heavily developed and will be continue to be!

Stephen Gates
@Stephen-Gates

this year was absolutely stellar for the whole Frictionless Data community.

@roll - I 100% agree and I'm looking forward to contributing in 2018.

Have you eliminated the idea of having goodtables-py/CLI as a backend for goodtables-js - frictionlessdata/goodtables-js#19

@roll I haven't eliminated this idea as we're not planning to implement data quality reports or advanced tests in Data Curator 2.0. I'd love to implement those features however our sponsor has other priorities for the first half of 2018.

Rufus Pollock
@rufuspollock

Last but not least I wish everyone Happy Merry Christmas (retrospectively=) and Happy New Year! I would say this year was absolutely stellar for the whole Frictionless Data community. The movement have finished the specs-v1 and implemented it for 9(!!!) languages. There are now goodtables.io stack, new-brand datahub.io, awesome projects like Data Curator and much more. It's really great to be a part of this project and community. :tada: :sparkles: :+1:

:thumbsup: :thumbsup: Happy Christmas and New Year everyone!

Rufus Pollock
@rufuspollock

@Stephen-Gates @roll i've booted an issue about test and exemplar data packages here frictionlessdata/implementations#28

@Stephen-Gates would you be up for contributing on this -- and anyone else out there :-)

Stephen Gates
@Stephen-Gates
@rufuspollock happy to contribute with some guidance on what’s needed. E.g. test for each data type, format, constraint, ...
Rufus Pollock
@rufuspollock
@Stephen-Gates that would be great - can you start commenting on the issue. Also if you know any locations of existing test data it would be great to list them.
Chris Hale
@chrispomeroyhale
Hi! I'm new here. I've been working independently on a Swift implementation of data package, tabular data package, and table schema. I presently have a draft implementation which is now in use in a shipping product. I would like to contribute (or collaborate with) this code if deemed suitable when I've reached a critical milestone. I had hoped I would reach that milestone by the end of the year but there's still much to do. Indeed, the present state of Swift creates some challenges for bonafide libraries. So, here’s to the gregorian new year!
Stephen Gates
@Stephen-Gates
@slythfox that sounds really interesting. What does your shipping product do with data packages? I’m sure others here can help you contribute. Happy New Year!
Chris Hale
@chrispomeroyhale
@Stephen-Gates My interest is in data liberation and longevity. So my application uses tabular data package as a kind of intermediary format for importing and exporting data stored in its relational database. With CoreData, I have implemented a custom persistent store for working with tabular data packages and CSV. I've also found it to be useful for seeding a database during development.
Stephen Gates
@Stephen-Gates
data liberation and longevity :+1:
On a different topic… In a table schema, what would be a reasonable maximum number of values to place in an enum constraint before it made more sense to create a foreign key relationship to a reference table? For context, I’m planning to implement features to create a constraint from column and create reference table from column.
Rufus Pollock
@rufuspollock

Hi! I'm new here. I've been working independently on a Swift implementation of data package, tabular data package, and table schema. I presently have a draft implementation which is now in use in a shipping product. I would like to contribute (or collaborate with) this code if deemed suitable when I've reached a critical milestone. I had hoped I would reach that milestone by the end of the year but there's still much to do. Indeed, the present state of Swift creates some challenges for bonafide libraries. So, here’s to the gregorian new year!

@slythfox this is great. Would you be up for opening an issue about your implementation work so far here https://github.com/frictionlessdata/implementations/issues so we can keep track of it.

It is also really really interesting to hear of the work you've been doing with the core datasets. There primary home how is https://datahub.io/core and https://datahub.io/docs/core-data

@Stephen-Gates

On a different topic… In a table schema, what would be a reasonable maximum number of values to place in an enum constraint before it made more sense to create a foreign key relationship to a reference table? For context, I’m planning to implement features to create a constraint from column and create reference table from column.

Good question. I'm not sure there is any obvious answer to that. Can i suggest opening a question on discuss.okfn.org on this as it seems kind of FAQ like and worth a more permanent response.

Stephen Gates
@Stephen-Gates

I'd like to progress integration between CKAN and Frictionless Data. I've been testing the CKAN Data Packager extension that raised a number of issues that I aggregated into a Minimum Viable Product issue frictionlessdata/ckanext-datapackager#63. I explored things further in this document and discovered a number of connections with CKAN Validator, CKAN Scheming, and prior thoughts on adding schemas to CKAN.

@Starl3n put out a Call for contributions and @dkelsey has offered to help :+1:

I think we can safely start on fixing the conversion of licenses, contributors, and sources properties for a data package.

I would really appreciate some guidance on tackling profile, schemas, dialect and encoding.

It seems that:

  • there should be some CKAN Schema for the Data Package properties that could be applied using CKAN Scheming
  • CKAN Validator is tackling storing Table Schemas in CKAN

Suggestions on how to progress would be greatly appreciated.

Stephen Gates
@Stephen-Gates
^^ Actually, before starting, I'd like some feedback on converting CKAN Author and Maintainer to contributors, and sources properties for a data package. See frictionlessdata/ckanext-datapackager#59
Chris Hale
@chrispomeroyhale

@rufuspollock Should I add my Swift implementation status in frictionlessdata/implementations#26 or were you wanting me to create a separate issue?

Neat stuff on DataHub! Although I was referring to Apple’s CoreData which is an object graph. I have a persistent store backend for importing/exporting between a SQLite store and a tabular data package.

Matthew Thompson
@cblop
Hi all, just making sure I keep you up to date on the Clojure implementation. I'm implementing the Table Schema with Clojure's Spec library for the types at the moment. Progress for the next month or so is going to come to a halt though, as I have to finish making corrections on my PhD thesis. Just leaving you this note to let you know that I am still very much working on it.
jobarratt
@jobarratt
@cblop thank you very much for the update. Happy New Year and good luck with the PhD
Rufus Pollock
@rufuspollock

@slythfox

@rufuspollock Should I add my Swift implementation status in frictionlessdata/implementations#26 or were you wanting me to create a separate issue?

I'd suggest opening a separate issue.

Matthew Thompson
@cblop
@jobarratt thanks Jo!