Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Ricardo Garcia Silva
    @ricardogsilva
    keeping it in config files (or as environment variables) makes it easier to automate installs
    lets think about it some more and try to come up with a spec for this
    Tom Kralidis
    @tomkralidis
    k
    Ricardo Garcia Silva
    @ricardogsilva
    in the wiki perhaps?
    Tom Kralidis
    @tomkralidis
    +1.
    ok, I'll take that action
    Ricardo Garcia Silva
    @ricardogsilva
    are mostly done then?
    Tom Kralidis
    @tomkralidis
    anything else to add? Next meeting in 4 weeks? Say 06 March?
    Ricardo Garcia Silva
    @ricardogsilva
    I can write up the minutes from the meeting, like last time
    yes, 06 march seems OK
    Tom Kralidis
    @tomkralidis
    thanks @ricardogsilva!
    thanks and have a good one ttyl
    Ricardo Garcia Silva
    @ricardogsilva
    bye bye
    happy sprinting!
    Tom Kralidis
    @tomkralidis
    thanks! let's try to move pyfes forward this week!
    Ricardo Garcia Silva
    @ricardogsilva
    fyi, notes from the second gitter meeting are available at https://github.com/geopython/pycsw/wiki/Meetings-2017-02-06 please review/edit them at your discretion
    Tom Kralidis
    @tomkralidis
    thanks @ricardogsilva can you send to pycsw-devel mailing list if you have time?
    Tom Kralidis
    @tomkralidis
    @ricardogsilva for pyfes I added a task to review existing implementations https://github.com/geopython/pyfes/projects/1#card-1627023
    there may be others
    Tom Kralidis
    @tomkralidis
    @ricardogsilva can we remove the requirements-pg.txt include in https://github.com/geopython/pycsw/blob/master/requirements-dev.txt#L3 ? It's tripping up non PG workflows for me. Do we need this in the dev requirements?
    Tom Kralidis
    @tomkralidis
    or is there some flag to 'turn off' PG based testing ?
    Angelos Tzotsos
    @kalxas
    hi all
    sorry for missing the meeting, crazy day, forgot about it
    will catch up from logs
    Tom Kralidis
    @tomkralidis
    hi all: great first day at code sprint; I had numerous pycsw discussions which I would summarize as requests for supporting multiple backends (ES/SOLR, other HTTP APIs, etc.). I would say pyfes is job 1 at this point.
    Ricardo Garcia Silva
    @ricardogsilva

    @tomkralidis not requiring requirements-pg.txt in requirements-dev.txt is not something I thought of while implementing it. My rationale was that usually tests would be run for all backends so I was expecting all dependencies to be installed.

    In order to be able to skip on requirements-pg.txt we'd need to have some sort of conditional import of psycopg2 in tests/functionaltests/conftest.py. It is doable, but requires a bit of additional work.

    If you just want to disable testing pg stuff when running tests, with tox you can run tox --env py27-sqlite --env py34-sqlite in a similar way as is done on pycsw's .travis.yml file. You'd still need to have requirements-pg.txt installed though...

    If this is a showstopper for you, please open a ticket and assign me to it

    Ricardo Garcia Silva
    @ricardogsilva
    @tomkralidis thanks for the research on other FES implementations, I'll look into them too, when I have the time. Did you read the architecture sketches page that I've posted on the wiki?
    Tom Kralidis
    @tomkralidis
    @ricardogsilva thanks for the info. The issue I ran into is that my dev env cannot install psycopg2 probably because I don't have underlying PG libs. So as a poor dev I need to be able to test without PG on my system :)
    Tom Kralidis
    @tomkralidis
    ok, filed: geopython/pycsw#497
    @ricardogsilva for pyges I had a look at https://github.com/geopython/pyfes/wiki/Architecture-sketches and have a bunch of q's
    Tom Kralidis
    @tomkralidis
    like what's the difference between a filter parser and a query parser?
    Ricardo Garcia Silva
    @ricardogsilva
    • A filter parser is a parser for the FILTER_LANGUAGE (using the KVP parameter name) specified in the query. In CSW we typically use two different filter languages: FES and CQL. So I guess we'd need a filter parser for FES stuff and another for CQL stuff in pyfes.

    • A query parser is a parser for the whole query. In the FES spec, a query is composed of (again using the KVP parameter names) TYPENAMES, PROPERTYNAMES, FILTER and SORTBY. So a query parser needs to know how to extract this stuff from the request. Also, a query parser may use more than one filter parser. This may happen for example in a hypothetical parser trying to parse a CSW GetRecords request, where either one of FES or CQL might be used as the filter language

    I guess this is rather dense... Am I making any sense to you?

    Tom Kralidis
    @tomkralidis
    it's dense territory :) we're getting there :)
    I spent a big part of yesterday thinking about the backends. Question is how much real filter processing does a backend need to do?
    Ricardo Garcia Silva
    @ricardogsilva
    can you elaborate on that?
    Tom Kralidis
    @tomkralidis
    e.g. currently pycsw.ogc.fes.fes1 parses the query into an SQL where clause. pyfes would abstract that away instead into some object/thing. So then would a backend need to crawl through the pyfes object with similar work to be able to construct the relevant query in the backend's space?
    Ricardo Garcia Silva
    @ricardogsilva
    yeah, I guess so. The rationale being that pyfes could abstract the actual query language in use (which could be FES, CQL or other) into a common query format
    this means that the backend still has to do the work of translating the query into its own domain
    Tom Kralidis
    @tomkralidis
    yeah. There is a slippery slope between that and a backend just processing the FES XML straight. But then again expecting a backend to process FES XML is heavy/won't scale.
    Ricardo Garcia Silva
    @ricardogsilva
    ideally we'd be able to make pyfes easier to use than directly parsing the XML (or other language) and thus more attractive
    Tom Kralidis
    @tomkralidis
    agree.
    so a backend would process a pyfes type?
    Ricardo Garcia Silva
    @ricardogsilva
    that is why I'd like to have a simple API for pyfes clients, with just using some parse(some_data) function that would take care of instatiating these more esoteric query and filter parser objects in the backseat and it would return the already digested fes type for the backend to process afterwards
    so something like:
    
    fes_stuff = pyfes.parser.parse(my_xml_data, **additional_arguments)
    records = repository.perform_pyfes_query(fes_stuff)
    or close to that
    Tom Kralidis
    @tomkralidis
    exactly
    Ricardo Garcia Silva
    @ricardogsilva
    those additional_aguments could be stuff to customize, e.g. some validation on ValueReference and Literal expression types, maybe a custom etree parser (hardened against XML attacks), etc
    Tom Kralidis
    @tomkralidis
    and the serializers to convert back?
    Ricardo Garcia Silva
    @ricardogsilva
    I think those would be more useful for clients of an OGC service (like owslib), where the user could specify a query by using CQL (as is the case in geotools) or by creating the fes types directly, and then serializing to XML before sending a request to the server