by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 27 15:01
    cnavacch commented #621
  • Jul 27 14:47
    tomkralidis commented #621
  • Jul 27 14:00
    cnavacch commented #621
  • Jul 27 11:28
    tomkralidis commented #621
  • Jul 26 23:10
    tomkralidis closed #620
  • Jul 26 23:10
    tomkralidis commented #620
  • Jul 24 13:40
    cnavacch opened #621
  • Jul 22 20:34
    kishkumar96 edited #620
  • Jul 22 20:34
    kishkumar96 edited #620
  • Jul 22 20:34
    kishkumar96 opened #620
  • Jul 15 09:15
    Rastopapola commented #619
  • Jul 15 08:49
    ricardogsilva commented #619
  • Jul 15 02:57
    tomkralidis assigned #619
  • Jul 15 02:16
    tomkralidis commented #619
  • Jun 13 12:13

    tomkralidis on master

    Create FUNDING.yml (compare)

  • Jun 02 00:52
    tomkralidis closed #618
  • May 31 10:46
    kalxas commented #572
  • May 19 18:04
    tomkralidis commented #572
  • May 19 15:32
    rsignell-usgs commented #572
  • May 19 15:31
    rsignell-usgs commented #572
Ricardo Garcia Silva
@ricardogsilva
keeping it in config files (or as environment variables) makes it easier to automate installs
lets think about it some more and try to come up with a spec for this
Tom Kralidis
@tomkralidis
k
Ricardo Garcia Silva
@ricardogsilva
in the wiki perhaps?
Tom Kralidis
@tomkralidis
+1.
ok, I'll take that action
Ricardo Garcia Silva
@ricardogsilva
are mostly done then?
Tom Kralidis
@tomkralidis
anything else to add? Next meeting in 4 weeks? Say 06 March?
Ricardo Garcia Silva
@ricardogsilva
I can write up the minutes from the meeting, like last time
yes, 06 march seems OK
Tom Kralidis
@tomkralidis
thanks @ricardogsilva!
thanks and have a good one ttyl
Ricardo Garcia Silva
@ricardogsilva
bye bye
happy sprinting!
Tom Kralidis
@tomkralidis
thanks! let's try to move pyfes forward this week!
Ricardo Garcia Silva
@ricardogsilva
fyi, notes from the second gitter meeting are available at https://github.com/geopython/pycsw/wiki/Meetings-2017-02-06 please review/edit them at your discretion
Tom Kralidis
@tomkralidis
thanks @ricardogsilva can you send to pycsw-devel mailing list if you have time?
Tom Kralidis
@tomkralidis
@ricardogsilva for pyfes I added a task to review existing implementations https://github.com/geopython/pyfes/projects/1#card-1627023
there may be others
Tom Kralidis
@tomkralidis
@ricardogsilva can we remove the requirements-pg.txt include in https://github.com/geopython/pycsw/blob/master/requirements-dev.txt#L3 ? It's tripping up non PG workflows for me. Do we need this in the dev requirements?
Tom Kralidis
@tomkralidis
or is there some flag to 'turn off' PG based testing ?
Angelos Tzotsos
@kalxas
hi all
sorry for missing the meeting, crazy day, forgot about it
will catch up from logs
Tom Kralidis
@tomkralidis
hi all: great first day at code sprint; I had numerous pycsw discussions which I would summarize as requests for supporting multiple backends (ES/SOLR, other HTTP APIs, etc.). I would say pyfes is job 1 at this point.
Ricardo Garcia Silva
@ricardogsilva

@tomkralidis not requiring requirements-pg.txt in requirements-dev.txt is not something I thought of while implementing it. My rationale was that usually tests would be run for all backends so I was expecting all dependencies to be installed.

In order to be able to skip on requirements-pg.txt we'd need to have some sort of conditional import of psycopg2 in tests/functionaltests/conftest.py. It is doable, but requires a bit of additional work.

If you just want to disable testing pg stuff when running tests, with tox you can run tox --env py27-sqlite --env py34-sqlite in a similar way as is done on pycsw's .travis.yml file. You'd still need to have requirements-pg.txt installed though...

If this is a showstopper for you, please open a ticket and assign me to it

Ricardo Garcia Silva
@ricardogsilva
@tomkralidis thanks for the research on other FES implementations, I'll look into them too, when I have the time. Did you read the architecture sketches page that I've posted on the wiki?
Tom Kralidis
@tomkralidis
@ricardogsilva thanks for the info. The issue I ran into is that my dev env cannot install psycopg2 probably because I don't have underlying PG libs. So as a poor dev I need to be able to test without PG on my system :)
Tom Kralidis
@tomkralidis
ok, filed: geopython/pycsw#497
@ricardogsilva for pyges I had a look at https://github.com/geopython/pyfes/wiki/Architecture-sketches and have a bunch of q's
Tom Kralidis
@tomkralidis
like what's the difference between a filter parser and a query parser?
Ricardo Garcia Silva
@ricardogsilva
  • A filter parser is a parser for the FILTER_LANGUAGE (using the KVP parameter name) specified in the query. In CSW we typically use two different filter languages: FES and CQL. So I guess we'd need a filter parser for FES stuff and another for CQL stuff in pyfes.

  • A query parser is a parser for the whole query. In the FES spec, a query is composed of (again using the KVP parameter names) TYPENAMES, PROPERTYNAMES, FILTER and SORTBY. So a query parser needs to know how to extract this stuff from the request. Also, a query parser may use more than one filter parser. This may happen for example in a hypothetical parser trying to parse a CSW GetRecords request, where either one of FES or CQL might be used as the filter language

I guess this is rather dense... Am I making any sense to you?

Tom Kralidis
@tomkralidis
it's dense territory :) we're getting there :)
I spent a big part of yesterday thinking about the backends. Question is how much real filter processing does a backend need to do?
Ricardo Garcia Silva
@ricardogsilva
can you elaborate on that?
Tom Kralidis
@tomkralidis
e.g. currently pycsw.ogc.fes.fes1 parses the query into an SQL where clause. pyfes would abstract that away instead into some object/thing. So then would a backend need to crawl through the pyfes object with similar work to be able to construct the relevant query in the backend's space?
Ricardo Garcia Silva
@ricardogsilva
yeah, I guess so. The rationale being that pyfes could abstract the actual query language in use (which could be FES, CQL or other) into a common query format
this means that the backend still has to do the work of translating the query into its own domain
Tom Kralidis
@tomkralidis
yeah. There is a slippery slope between that and a backend just processing the FES XML straight. But then again expecting a backend to process FES XML is heavy/won't scale.
Ricardo Garcia Silva
@ricardogsilva
ideally we'd be able to make pyfes easier to use than directly parsing the XML (or other language) and thus more attractive
Tom Kralidis
@tomkralidis
agree.
so a backend would process a pyfes type?
Ricardo Garcia Silva
@ricardogsilva
that is why I'd like to have a simple API for pyfes clients, with just using some parse(some_data) function that would take care of instatiating these more esoteric query and filter parser objects in the backseat and it would return the already digested fes type for the backend to process afterwards
so something like:

fes_stuff = pyfes.parser.parse(my_xml_data, **additional_arguments)
records = repository.perform_pyfes_query(fes_stuff)
or close to that
Tom Kralidis
@tomkralidis
exactly
Ricardo Garcia Silva
@ricardogsilva
those additional_aguments could be stuff to customize, e.g. some validation on ValueReference and Literal expression types, maybe a custom etree parser (hardened against XML attacks), etc
Tom Kralidis
@tomkralidis
and the serializers to convert back?
Ricardo Garcia Silva
@ricardogsilva
I think those would be more useful for clients of an OGC service (like owslib), where the user could specify a query by using CQL (as is the case in geotools) or by creating the fes types directly, and then serializing to XML before sending a request to the server