These are chat archives for frictionlessdata/chat

1st
Apr 2017
roll
@roll
Apr 01 2017 06:44
@esjewett I've added an issue - frictionlessdata/specs#401
roll
@roll
Apr 01 2017 08:00
@pwalsh @rufuspollock @Stiivi XML Schema uses a term lexical representation - may be that's a missing part for Table Schema? As I was saying above in our type definitions we are kinda liberal in talking about both at the same time - native/cast/typed values and value string representations. E.g. for year - year must be an integer from 0 to 9999 and as a lexical representation must contain 4 digits.
Ori Hoch
@OriHoch
Apr 01 2017 08:45
I'm trying to specify a date field in json table schema - it seems there is inconsistency between specs and current implementations (e.g. tableschema-py)
It seems that implementations required it to be specified like "fmt:%Y-%m-%d"
but in the standard it's not clear how it should be specified, should it be wrapped in curly brackets like "{%y-%m-%d}" ? or without curly brackets?
also, how should I handle it at the moment? Are you intending to support backwards compatibility? Or will I need to modify all our schemas when switching to the new version?
Thanks..
roll
@roll
Apr 01 2017 09:25
@OriHoch Also nice to meet you. My name is Evgeny I'm a tech lead for Frictionless Data core implementations. I was kinda keeping a low profile in this channel lately but just because I was working hard to update tableschema-py to specs-v1 :smile: And make it much more ready to be a role model for porting spec to other languages (after the mentioned PR merge only validate and infer functions will be not fully reviewed and polished). So feel free to reach me on any questions about tools implementation.
Ori Hoch
@OriHoch
Apr 01 2017 09:30
thanks, I understand that you don't intend to provide backwards compatibility? so all schemas will need to change to match v1 to use v1 implementations?
roll
@roll
Apr 01 2017 09:34
Yes tableschema-py-v1 will be working only with spec-v1 compliant schemas. Of course for new language implementations only spec-v1 matters. So for us now this - http://specs.frictionlessdata.io/table-schema/ - is the only one point of truth. Exception could be made for datapackage.path/url but lets discuss it separately.
Rufus Pollock
@rufuspollock
Apr 01 2017 09:35
@roll can we try some backwards compatibility? I have explicitly mentioned this several times re the change and in previous spec changes … - it is quite a big deal and analysing what it would take for this would be useful and would help re e.g. pandas integration etc0
roll
@roll
Apr 01 2017 09:36
@rufuspollock yes of course - just let's do use case-basis
Rufus Pollock
@rufuspollock
Apr 01 2017 09:36
@pwalsh ^^^ - backwards compatibility is a big deal and would mesh with the “loose” vs “strict” idea at this point.
@roll agreed - even producing a simple spreadsheet / doc of changes would be smart right now … e.g. what are the key changes so implementors can know and we can guide users.
Ori Hoch
@OriHoch
Apr 01 2017 09:41
Currently we are writing some schemas which we are using in hasadna projects, which are using pre-v1 tools
It would be great to have backwards compatibility to allow them to continue to work with newer tool versions..
Perhaps an optional backwards compatibility layer or a library that will take an old schema and convert it to new schema - I guess something like that will be useful for us
Rufus Pollock
@rufuspollock
Apr 01 2017 09:42
@OriHoch :thumbsup: - and as one of the co-editors of the specs with @pwalsh and others you can know this is something we’ll really look at ...
roll
@roll
Apr 01 2017 09:43
@rufuspollock for example datapackage.resource.url/path should be first backward-comp thing. Let's enhance this list during time. But I suppose we just don't need to go to much on this road because v1 is important milestone. Which should bring stability and too many deprecated things are still instability because publishers will be put to work with schemas which is not spec-v1-valid
@OriHoch Let's see on concrete examples. Problem here that not libs but pre-v1 and v1 specs is mutually exclusive in some aspects.
Rufus Pollock
@rufuspollock
Apr 01 2017 09:45

@roll yes on first point no on second ;-)

. But I suppose we just don't need to go to much on this road because v1 is important milestone.

That’s not true here, we have thousands of data package in the wild and user experience and an obsession with making things great for publisehrs and consumers is part of our DNA here. Preserving some form of compatability (or at least clear and easy migration pathway) really matters for users — and is crucial to ensuring current momentum is not disrupted.

Ori Hoch
@OriHoch
Apr 01 2017 09:48
Currently I saw it on the date table schema field where strftime had to be prefixed with fmt: and in new spec it doesn't need the prefix
this can easily be fixed with a backward compat layer
For example - having a LegacyField class extending the new Field class - and doing these conversions
roll
@roll
Apr 01 2017 09:48
@rufuspollock I mean an idea like cover 20% of breaking changes that the most important (80%) and provide migration guide (as you said) to minor things
Ori Hoch
@OriHoch
Apr 01 2017 09:49
also, need to consider that people might not specify package version in their requirements - so things will break when v1 is released
Rufus Pollock
@rufuspollock
Apr 01 2017 09:49
@roll :thumbsup: @OriHoch :thumbsup:
roll
@roll
Apr 01 2017 09:52
@OriHoch for now tableschema-py is released on PyPi as v1.0.0-alpha1so it can't be installed without a flag pip install tableschema --pre. But yes when we'll be on final release we should be ready. Also I'm going to clearly describe in libs how we follow sevmer (now it's only badge=) and how libs should be used in requirements.txt e.g. tableschema < 2.0.0 after v1 release which will guarantee no breaking changes.
@pwalsh @rufuspollock I think we need to bootstrap some document/issue with ordered list of most wanted backward-comp features to implement
Paul Walsh
@pwalsh
Apr 01 2017 18:25

@OriHoch I agree we need to handle old descriptors that use fmt: and the support should be trivial.

@rufuspollock the amount if code required for backwards compat support should be quite straight forward. I’m aware of all the changes and closely watching the python work for v1 support so I can track this and identify areas appropriately.

Adam Kariv
@akariv
Apr 01 2017 18:50
What about specifying in the data package itself the spec version it's compatible with? When the next version arrives, the libraries will need to know how to validate and parse datapackages from all versions. It would be easier if the spec version was explicitly stated inside the descriptor.
Paul Walsh
@pwalsh
Apr 01 2017 19:35

@akariv we currently have profile, which is like specification metadata. I’m working on a PR now to have specification metadata under spec and support

'spec’: {
  ‘profile': 'tabular-data-package’,
  ‘version': '1.0'
}

etc.