geopython/GeoHealthCheck

Service Status Checker for OGC Web Services

Just van den Broecke

@justb4

@epifanio yes, it should give you a fresh start. You can remove the Docker ghc_pgdb Volume, with docker volume rm when all stopped/Containers removed.

epifanio

@epifanio

I will try no, just a check on my configuration .. in my docker-compose I have:

  ghc_web:
    image: epinux/ghck:latest

    restart: unless-stopped

    env_file:
      - ghck.env
      - ghck-postgis.env

so, both env files have DB instructuons:

ghc.env:

SQLALCHEMY_DATABASE_URI=sqlite:////GeoHealthCheck/DB/data.db

ghck-postgis.env
```
# We mainly need to override the default (sqlite) DB URI
SQLALCHEMY_DATABASE_URI=postgresql://ghc:ghc@postgis_ghck:5432/ghc
```
I guess this is wrong and maybe is that the cause of the error?
I mean should the DB both point to postgres, correct?

or it will be overrided .. and I should not care?

Just van den Broecke

@justb4

Yes, but you only need to have SQLALCHEMY_DATABASE_URI defined once, e.g. in ghc.env. ghck-postgis.env is probably only needed for the postgis Container in your Compose file. It probably only needs to contain POSTGRES_* vars.

Just van den Broecke

@justb4

Starting GHC afresh with a new DB with Docker won't work at the moment, see #396. Working on a quickfix...

epifanio

@epifanio

thanks! monitoringthe logs!

Just van den Broecke

@justb4

#396 solved. No errors when starting Docker (or a DB upgrade) from a new or existing DB.

Still a major issue: #395 which puzzles. Probably has to do with 'last_run'...

Just van den Broecke

@justb4

#395 solved via PR #398. Luckily no revert needed.

epifanio

@epifanio

testing Ø= !

Just van den Broecke

@justb4

@epifanio sharing a Python wisdom? Guess you are fine with #398?

jochemthart1

@jochemthart1

@justb4
For the WMTS probe, I would like to be able to specify if the resource should be accessed through Rest or KVP. This is necessary when the base url is the same for both the Rest and KVP endpoint like with the following endpoints "https://host/wmts/1.0.0/WMTSCapabilities.xml" or https://host/wmts?service=WMTS......". In GeoHealthCheck you are supposed to enter only the base url, in this case: "https://host/wmts". If I want to check both endpoints, I would like to add them as 2 seperate resources.

To solve this problem I can think of 2 solutions:

Enter the entire GetCapabilities resource url, which ensures that a Rest or KVP endpoint can be distinguished.
Make Rest or KVP a user-input parameter in the PARAM_DEFS dictionary inside the probe. This user input can be automatically defined by choosing KVP as standard, except if the owslib.wmts.WebMapTileService object contains the attribute restonly=True.

Which solution has your preference and/or do you know of another solution?

Just van den Broecke

@justb4

Hmm, tricky question. At least not all implementations support both KVP and REST (I remember PDOK 'Luchtfoto' at the time, KVP-only).
Option 1) does not follow the common GHC convention: to enter only the endpoint URL, not the Caps URL.
I rather see one Resource as one item/endpoint. So leaves option 2). Is it an idea to enter only the base url, in this case: "https://host/wmts" and to have three choices in the Resource PARAM_DEF: KVP, REST, Both? In the latter case both options are tested, within a single Run?

epifanio

@epifanio

trying to access a freshly build .. I have the service running on https://ghc.epinux.com/ in my swarmpit user interface I can see the ENV of the running service, like:

GHC_ADMIN_EMAIL
myemail@gmail.com

GHC_ADMIN_NAME
admin

GHC_ADMIN_PWD
mypassword

but when attemting to log-in it says "Invalid username and / or password"

I checked in the postgis container, the table users with psql:

ghc=# SELECT * FROM user;
 user 
------
 ghc
(1 row)

How do I log in? :(

epifanio

@epifanio

where can I change admin password in the DB ?

Just van den Broecke

@justb4

@epifanio GHC stores encrypted passwords in the user table. Even as an admin one cannot decrypt user passwords. What it looks like is that SELECT * FROM user shows the Postgres User table. Looks like you are in the Postgres DB. The GHC user table is in public.user but you must use psql to login into that DB. First make sure you have initialized a GHC DB in Postgres. See also the documentation on GHC password management.

epifanio

@epifanio

@justb4 thanks! logging in and check if I can update the admin password

the testing endpoint is at https://ghck.epinux.com

Just van den Broecke

@justb4

@epifanio Looks good, registered endpoints! Yes you can update the password as in doc above. Or reset via email if configured ok.

epifanio

@epifanio

yes, still playing with the config. I see it has 'locaalhost' hardcoded in the mail message .. looking into it now

Just van den Broecke

@justb4

You must configure an SMTP endpoint for that, also to get notifications by email. Some use a gmail account (be sure to allow "untrusted" apps in gmail config).

epifanio

@epifanio

yes, I already have it set up this way, using an app-passwd from gmail.
the email that I receive is showing the correct value for the test, but has the localhost variable in it - I guess it comes from the
GHC_SITE_URL=''
value from the dockerfile. I am now overriding it in the ghc.env

epifanio

@epifanio

the relevant part of the log: (Background on this error at: http://sqlalche.me/e/e3q8)

epifanio

@epifanio

I added to whole log as a gist here: https://gist.github.com/epifanio/f4926f00e2893b8ab1c80b59611dbab6

epifanio

@epifanio

I can't find in the code where the db connection is creatred (usually 'create_engine' in sqlalchemy) but seems such call is hidden behind flask_sqlalchemy

I was trying to implement : https://docs.sqlalchemy.org/en/14/core/pooling.html#disconnect-handling-pessimistic

best guess is to use this: https://flask-sqlalchemy.palletsprojects.com/en/2.x/api/?highlight=create_engine#flask_sqlalchemy.SQLAlchemy.create_engine

epifanio

@epifanio

How to tell the flask
_sqlalchemy to read the following option?

    SQLALCHEMY_ENGINE_OPTIONS = {
                                 'pre_ping': True
                                 }

Just van den Broecke

@justb4

From what I see in the log there is a DB failure on the SELECT on Resource table, never any other query like the ResourceLock table:

SELECT resource.auth AS resource_auth, resource.identifier AS resource_identifier, resource.resource_type AS resource_resource_type, resource.active AS resource_active, resource.title AS resource_title, resource.url AS resource_url, resource.latitude AS resource_latitude, resource.longitude AS resource_longitude, resource.owner_identifier AS resource_owner_identifier, resource.run_frequency AS resource_run_frequency FROM resource

I never see these errors on demo.geohealthcheck.org, which runs the latest Docker Image. Are you running with Docker and the latest version?

Yes, GHC_SITE_URL determines the, URL mentioned in emails.

epifanio

@epifanio

Yes, I am running it with docker - using the geopython docker image (latest built)
mysettings is to run GHCK in a docker swarm proxied with traefik

we experencied a similar issue with pycsw with the same settings

it went away using this approach geopython/pycsw#696

and this geopython/pycsw#685

epifanio

@epifanio

you can test it by loading the service at this url: https://ghck.epinux.com/ i will leave it untouched for a while, the internal server error happes when the app is cold nooane using it ..

Just van den Broecke

@justb4

Interesting, loosing DB-connections (and reconnecting) is often an issue with databases. The Docker startup script does something familiar in the configure.sh.
It looks like these can be configured in Flask_SQLAlchemy, pssobly in the standard GHC config file. But how to pass these to the Docker Containers is a bit a puzzle, also as SQLALCHEMY_ENGINE_OPTIONS is a dict. But as more and more deploys will use cloud technology like Kubernetes and Swarm we must solve this. PR welcome!

Just van den Broecke

@justb4

In init.py the SQLAlchemy object is created which has an engine_options parameter.

epifanio

@epifanio

i will start adding the pre-ping as option to the engine and see how it behavies

Just van den Broecke

@justb4

Ok, great!

epifanio

@epifanio

testingthe changes on https://ghck.epinux.com/ it seems there is no longer internal server error.

epifanio

@epifanio

@justb4 the PR should fix the internal server error I was having. One qestion: I first started the envirnment without setting the notification email addresses -- so it was using the default value of: admin.istrator@mydomain.com
I then restarted the service using a configured environment with a proper list of email that will receive the notification.
I noticed that the old email address: admin.istrator@mydomain.com is still used by the system. I guess its entry didn't get removed from the db

Just van den Broecke

@justb4

Thanks @epifanio for PR #400. I added my review. Main question is: how much overhead adds pre_ping, as the engine/dialect will issue a SELECT 1 for each and every DB-access, at least when accessing a DB-connection. Discussion in PR. But the feature is very useful, e.g. also to survive DB restarts.

epifanio

@epifanio

Thanks for you comment @justb4 I added a reply

I agree having it as a configuarble makes things more flexible

i can amend the PR according with the comment

epifanio

@epifanio

@justb4 how does it look geopython/GeoHealthCheck#400 ?

Just van den Broecke

@justb4

@epifanio see my review there.

epifanio

@epifanio

@justb4 great! thanks, I will work on it :)

Where communities thrive

People

Repo info

Activity