geopython/pycsw

pycsw is an OGC CSW server implementation written in Python.

Ricardo Garcia Silva

@ricardogsilva

keeping it in config files (or as environment variables) makes it easier to automate installs

lets think about it some more and try to come up with a spec for this

Tom Kralidis

@tomkralidis

Ricardo Garcia Silva

@ricardogsilva

in the wiki perhaps?

Tom Kralidis

@tomkralidis

+1.

ok, I'll take that action

Ricardo Garcia Silva

@ricardogsilva

are mostly done then?

Tom Kralidis

@tomkralidis

anything else to add? Next meeting in 4 weeks? Say 06 March?

Ricardo Garcia Silva

@ricardogsilva

I can write up the minutes from the meeting, like last time

yes, 06 march seems OK

Tom Kralidis

@tomkralidis

thanks @ricardogsilva!

thanks and have a good one ttyl

Ricardo Garcia Silva

@ricardogsilva

bye bye

happy sprinting!

Tom Kralidis

@tomkralidis

thanks! let's try to move pyfes forward this week!

Ricardo Garcia Silva

@ricardogsilva

fyi, notes from the second gitter meeting are available at https://github.com/geopython/pycsw/wiki/Meetings-2017-02-06 please review/edit them at your discretion

Tom Kralidis

@tomkralidis

thanks @ricardogsilva can you send to pycsw-devel mailing list if you have time?

Tom Kralidis

@tomkralidis

@ricardogsilva for pyfes I added a task to review existing implementations https://github.com/geopython/pyfes/projects/1#card-1627023

there may be others

Tom Kralidis

@tomkralidis

@ricardogsilva can we remove the requirements-pg.txt include in https://github.com/geopython/pycsw/blob/master/requirements-dev.txt#L3 ? It's tripping up non PG workflows for me. Do we need this in the dev requirements?

Tom Kralidis

@tomkralidis

or is there some flag to 'turn off' PG based testing ?

Angelos Tzotsos

@kalxas

hi all

sorry for missing the meeting, crazy day, forgot about it

will catch up from logs

Tom Kralidis

@tomkralidis

hi all: great first day at code sprint; I had numerous pycsw discussions which I would summarize as requests for supporting multiple backends (ES/SOLR, other HTTP APIs, etc.). I would say pyfes is job 1 at this point.

Ricardo Garcia Silva

@ricardogsilva

@tomkralidis not requiring requirements-pg.txt in requirements-dev.txt is not something I thought of while implementing it. My rationale was that usually tests would be run for all backends so I was expecting all dependencies to be installed.

In order to be able to skip on requirements-pg.txt we'd need to have some sort of conditional import of psycopg2 in tests/functionaltests/conftest.py. It is doable, but requires a bit of additional work.

If you just want to disable testing pg stuff when running tests, with tox you can run tox --env py27-sqlite --env py34-sqlite in a similar way as is done on pycsw's .travis.yml file. You'd still need to have requirements-pg.txt installed though...

If this is a showstopper for you, please open a ticket and assign me to it

Ricardo Garcia Silva

@ricardogsilva

@tomkralidis thanks for the research on other FES implementations, I'll look into them too, when I have the time. Did you read the architecture sketches page that I've posted on the wiki?

Tom Kralidis

@tomkralidis

@ricardogsilva thanks for the info. The issue I ran into is that my dev env cannot install psycopg2 probably because I don't have underlying PG libs. So as a poor dev I need to be able to test without PG on my system :)

Tom Kralidis

@tomkralidis

ok, filed: geopython/pycsw#497

@ricardogsilva for pyges I had a look at https://github.com/geopython/pyfes/wiki/Architecture-sketches and have a bunch of q's

Tom Kralidis

@tomkralidis

like what's the difference between a filter parser and a query parser?

Ricardo Garcia Silva

@ricardogsilva

A filter parser is a parser for the FILTER_LANGUAGE (using the KVP parameter name) specified in the query. In CSW we typically use two different filter languages: FES and CQL. So I guess we'd need a filter parser for FES stuff and another for CQL stuff in pyfes.
A query parser is a parser for the whole query. In the FES spec, a query is composed of (again using the KVP parameter names) TYPENAMES, PROPERTYNAMES, FILTER and SORTBY. So a query parser needs to know how to extract this stuff from the request. Also, a query parser may use more than one filter parser. This may happen for example in a hypothetical parser trying to parse a CSW GetRecords request, where either one of FES or CQL might be used as the filter language

I guess this is rather dense... Am I making any sense to you?

Tom Kralidis

@tomkralidis

it's dense territory :) we're getting there :)

I spent a big part of yesterday thinking about the backends. Question is how much real filter processing does a backend need to do?

Ricardo Garcia Silva

@ricardogsilva

can you elaborate on that?

Tom Kralidis

@tomkralidis

e.g. currently pycsw.ogc.fes.fes1 parses the query into an SQL where clause. pyfes would abstract that away instead into some object/thing. So then would a backend need to crawl through the pyfes object with similar work to be able to construct the relevant query in the backend's space?

Ricardo Garcia Silva

@ricardogsilva

yeah, I guess so. The rationale being that pyfes could abstract the actual query language in use (which could be FES, CQL or other) into a common query format

this means that the backend still has to do the work of translating the query into its own domain

Tom Kralidis

@tomkralidis

yeah. There is a slippery slope between that and a backend just processing the FES XML straight. But then again expecting a backend to process FES XML is heavy/won't scale.

Ricardo Garcia Silva

@ricardogsilva

ideally we'd be able to make pyfes easier to use than directly parsing the XML (or other language) and thus more attractive

Tom Kralidis

@tomkralidis

agree.

so a backend would process a pyfes type?

Ricardo Garcia Silva

@ricardogsilva

that is why I'd like to have a simple API for pyfes clients, with just using some parse(some_data) function that would take care of instatiating these more esoteric query and filter parser objects in the backseat and it would return the already digested fes type for the backend to process afterwards

so something like:


fes_stuff = pyfes.parser.parse(my_xml_data, **additional_arguments)
records = repository.perform_pyfes_query(fes_stuff)

or close to that

Tom Kralidis

@tomkralidis

exactly

Ricardo Garcia Silva

@ricardogsilva

those additional_aguments could be stuff to customize, e.g. some validation on ValueReference and Literal expression types, maybe a custom etree parser (hardened against XML attacks), etc

Tom Kralidis

@tomkralidis

and the serializers to convert back?

Ricardo Garcia Silva

@ricardogsilva

I think those would be more useful for clients of an OGC service (like owslib), where the user could specify a query by using CQL (as is the case in geotools) or by creating the fes types directly, and then serializing to XML before sending a request to the server

Where communities thrive

People

Repo info

Activity