Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Matthias Mohr
    @m-mohr
    Welcome to the Validation room for the Python and Node Validation tools:
    Rob Emanuele
    @lossyrob
    PySTAC just released validation functionality for validating PySTAC objects (1.0.0-beta.2) as well as JSON for STAC 0.8 or later.
    Matthias Mohr
    @m-mohr
    James Banting
    @jbants
    @lossyrob Does the pystac validator work with URLs and can you set versions to validate against?
    Matthias Mohr
    @m-mohr
    Why's the latter needed by the way? Shouldn't that be retrieved from the stac_version field dynamically?
    Rob Emanuele
    @lossyrob
    https://github.com/azavea/pystac/blob/develop/pystac/validation/__init__.py#L46 lets you validate a dict with versions >= 0.8. You can supply the version or it will identify it based on the dict
    James Banting
    @jbants
    stac_version wasn’t added until 0.7 I believe?
    Oh cool
    Matthias Mohr
    @m-mohr
    Oh, I see... never came I because I don't support those old versions ;-)
    Rob Emanuele
    @lossyrob
    The identification logic will be able to figure out the stac version (or at least tries its best) if stac_version is missing
    James Banting
    @jbants
    perfect
    Frederico Liporace
    @fredliporace
    @lossyrob I'm using pystac validator (validate_dict(...)). The returned list are the schemas loaded and used for validation, is that correct? My sample uses two extensions, view and proj. but it seems only the view extension is being loaded:
    f = open('/home/liporace/github/cbers-2-stac/test/CBERS_4_MUX_20170528_090_084_L2.json'); js = json.load(f); validate_dict(js)
    ['https://schemas.stacspec.org/v1.0.0-beta.2/item-spec/json-schema/item.json', 'https://schemas.stacspec.org/v1.0.0-beta.2/extensions/view/json-schema/schema.json']
    Rob Emanuele
    @lossyrob
    proj is not the extension ID, it's projection, so that's where it's getting tripped up
    Frederico Liporace
    @fredliporace
    Thanks, that was the problem.
    Matthias Mohr
    @m-mohr
    New version of the node validator released.
    Matthias Mohr
    @m-mohr
    Hey @jbants and @lossyrob! I started a comparison for STAC validation tools: https://github.com/m-mohr/stac-node-validator/blob/master/COMPARISON.md
    Not meant for competition, but to let users find better what they need. I tried to be unopinionated and don't want to promote mine or so.
    Could you check whether something is wrong or missing? Any idea what categories we could add? :-)
    17 replies
    Jonathan Healy
    @jonhealy1
    Hi just an update. I have brought validation from pystac into stac-validator. Stac-validator handles errors with some nice output. I also set up click to start creating a cli-based stac-utility program. I am looking for ideas on what other functions the utility program could perform?
    Rob Emanuele
    @lossyrob
    @jonhealy1 that's awesome! I was working on setting up a click CLI too. I can push up what I have in a bit - just have placeholder commands - to the stac-utils github org and make you a committer if that would be helpful for collab
    3 replies
    Matthias Mohr
    @m-mohr
    Nice, @jonhealy1! Could you have a look at the comparison above and check whether that's up to date with your changes? :-)
    11 replies
    Rob Emanuele
    @lossyrob
    @jonhealy1 I'm thinking of refactoring the stactools codebase to implement a namespace package so that we can have independently installable components that have their own dependencies. This is dropping out of a conversation we had this morning in the zoom about wanting tools to convert data sources like landsat/planet/sentinel into STACs - so we could have a subpackage for each of those providers. I can move the cli into it's own submodule, and include a stactools root package that depends on it by default so if someone runs pip install stactools (once we publish) it will install the CLI by default. Also this would enable a plugin system that could add subcommands for any subpackages (published by us or made custom by someone else) that added functionality to the tools. Any thoughts/concerns about that?
    10 replies
    Frederico Liporace
    @fredliporace
    Hi, the cbers 4/4a stac 1.0 documents are available from https://stac.amskepler.com/v10/ . As I mentioned in the sprint this is a 0.7 API that serves 1.0 content. I'll review the API changes in the following days and update the API.
    Frederico Liporace
    @fredliporace
    @jbants I'm getting a validation error for https://cbers-stac-1-0.s3.amazonaws.com/CBERS4A/MUX/201/137/CBERS_4A_MUX_20200808_201_137_L4.json in STACLint, it complains about description being a mandatory field. I assume this a reference to the descrption field in assets, is that correct? This field is not required in the current spec.
    2 replies
    Matthias Mohr
    @m-mohr
    @fredliporace FYI: STAC Node Validator doesn't complain ;-)
    13 replies
    James Banting
    @jbants
    We are pushing a new python validator this week @fredliporace
    1 reply
    Chris Holmes
    @cholmes
    Hey @fredliporace - any chance you could enable CORS on the new catalog? Then https://stacindex.org/collections/cbers#/?t=catalogs can work without a proxy.
    7 replies
    Also would you be up to give a lightning talk on CBERS next week? If so please submit at https://forms.gle/YzjTPioB6TkmEcwZ8
    2 replies
    Frederico Liporace
    @fredliporace
    Some suggestions for the validation crowd based on my own errors....
    Flag unknown extensions identifiers: I mistakenly used 'proj' instead of 'projection' one time and got bitten twice: not only the extension id was wrong but I thought that the extension fields were validated when in fact they were ignored.
    Check for extension scope when possible. I was incorrectly using the eo extension at collection level - the node validator caught that special case but I think that was 'luck', if the extension did not define required fields maybe the check would pass.
    Matthias Mohr
    @m-mohr
    I think the Node Validator handles your cases.
    But some things are not easy to achive just with JSON Schemas. It's not clear from the schemas what the extension scope is and hardcoding a list of extensions is not very clean and future-proof
    Frederico Liporace
    @fredliporace
    I used the node validator last, so it is possible... Yes, some checks would have to be implemented by checks additional to json schema. Are you sure about my second point, my guess is that it was flagged by chance because of a mandatory field not being defined.... would it be flagged if there were no required fields in the used extension?
    Matthias Mohr
    @m-mohr
    Well if there's no field at all, it wouldn't be caught, but then the schema is a bit strange.
    In other cases we usually have a required properties field for items, which is not available in collections/catalogs, so would fail. In most other extensions we try to set minProperties:1 if there's no required field. So in most cases it should fail if the scope is not correct, but I can't guarantee it for all. But I'd more see that as a schema issue then...
    A schema that succeeds on an empty json would always be strange... ;-)
    This issue radiantearth/stac-spec#889 would make validation a bit more bound to scopes, but would also need additions in all schemas.
    Frederico Liporace
    @fredliporace

    ... we try to set minProperties:1 if there's no required field ...

    OK then

    Jonathan Healy
    @jonhealy1
    I updated staclint.com to work with 1.0.0-beta.2 today. We also recently released a new version of stac_validator.
    Chris Holmes
    @cholmes
    Nice!
    Rob Emanuele
    @lossyrob
    :tada:
    Jonathan Healy
    @jonhealy1
    @lossyrob Thanks for the pystac validation stuff. It helped a lot.
    Rob Emanuele
    @lossyrob
    for sure! Happy it helped, looking forward to continuing to improve it over time. If there’s anything that could change or be added that you’ve found working with it, let me know!