Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 25 19:16
    epifanio review_requested #400
  • Sep 25 19:15
    epifanio synchronize #400
  • Sep 25 11:59
    justb4 milestoned #402
  • Sep 25 11:59
    justb4 commented #401
  • Sep 25 11:58
    justb4 labeled #402
  • Sep 25 11:58
    justb4 opened #402
  • Sep 25 11:32

    tomkralidis on trailing-slashes

    (compare)

  • Sep 25 11:31

    justb4 on master

    Support trailing slashes on Fla… (compare)

  • Sep 25 11:31
    justb4 closed #401
  • Sep 25 11:30
    justb4 commented #401
  • Sep 25 11:05
    justb4 milestoned #401
  • Sep 25 11:05
    justb4 labeled #401
  • Sep 25 11:05
    justb4 assigned #401
  • Sep 24 11:30
    tomkralidis commented #401
  • Sep 24 00:48
    tomkralidis review_requested #401
  • Sep 24 00:48
    tomkralidis opened #401
  • Sep 24 00:42

    tomkralidis on trailing-slashes

    support trailing slashes on Fla… (compare)

  • Sep 21 10:34

    tomkralidis on master

    Update SECURITY.md (compare)

  • Sep 16 13:28
    epifanio review_requested #400
Just van den Broecke
@justb4
@epifanio yes, it should give you a fresh start. You can remove the Docker ghc_pgdb Volume, with docker volume rm when all stopped/Containers removed.
epifanio
@epifanio

I will try no, just a check on my configuration .. in my docker-compose I have:

  ghc_web:
    image: epinux/ghck:latest

    restart: unless-stopped

    env_file:
      - ghck.env
      - ghck-postgis.env

so, both env files have DB instructuons:

  • ghc.env:
    SQLALCHEMY_DATABASE_URI=sqlite:////GeoHealthCheck/DB/data.db
  • ghck-postgis.env
    # We mainly need to override the default (sqlite) DB URI
    SQLALCHEMY_DATABASE_URI=postgresql://ghc:ghc@postgis_ghck:5432/ghc
    I guess this is wrong and maybe is that the cause of the error?
    I mean should the DB both point to postgres, correct?
or it will be overrided .. and I should not care?
Just van den Broecke
@justb4
Yes, but you only need to have SQLALCHEMY_DATABASE_URI defined once, e.g. in ghc.env. ghck-postgis.env is probably only needed for the postgis Container in your Compose file. It probably only needs to contain POSTGRES_* vars.
Just van den Broecke
@justb4
Starting GHC afresh with a new DB with Docker won't work at the moment, see #396. Working on a quickfix...
epifanio
@epifanio
thanks! monitoringthe logs!
Just van den Broecke
@justb4
#396 solved. No errors when starting Docker (or a DB upgrade) from a new or existing DB.
Still a major issue: #395 which puzzles. Probably has to do with 'last_run'...
Just van den Broecke
@justb4
#395 solved via PR #398. Luckily no revert needed.
epifanio
@epifanio
testing Ø= !
:)
Just van den Broecke
@justb4
@epifanio sharing a Python wisdom? Guess you are fine with #398?
jochemthart1
@jochemthart1

@justb4
For the WMTS probe, I would like to be able to specify if the resource should be accessed through Rest or KVP. This is necessary when the base url is the same for both the Rest and KVP endpoint like with the following endpoints "https://host/wmts/1.0.0/WMTSCapabilities.xml" or https://host/wmts?service=WMTS......". In GeoHealthCheck you are supposed to enter only the base url, in this case: "https://host/wmts". If I want to check both endpoints, I would like to add them as 2 seperate resources.

To solve this problem I can think of 2 solutions:

  1. Enter the entire GetCapabilities resource url, which ensures that a Rest or KVP endpoint can be distinguished.
  2. Make Rest or KVP a user-input parameter in the PARAM_DEFS dictionary inside the probe. This user input can be automatically defined by choosing KVP as standard, except if the owslib.wmts.WebMapTileService object contains the attribute restonly=True.

Which solution has your preference and/or do you know of another solution?

Just van den Broecke
@justb4
Hmm, tricky question. At least not all implementations support both KVP and REST (I remember PDOK 'Luchtfoto' at the time, KVP-only).
Option 1) does not follow the common GHC convention: to enter only the endpoint URL, not the Caps URL.
I rather see one Resource as one item/endpoint. So leaves option 2). Is it an idea to enter only the base url, in this case: "https://host/wmts" and to have three choices in the Resource PARAM_DEF: KVP, REST, Both? In the latter case both options are tested, within a single Run?
epifanio
@epifanio

trying to access a freshly build .. I have the service running on https://ghc.epinux.com/ in my swarmpit user interface I can see the ENV of the running service, like:

GHC_ADMIN_EMAIL
myemail@gmail.com

GHC_ADMIN_NAME
admin

GHC_ADMIN_PWD
mypassword

but when attemting to log-in it says "Invalid username and / or password"

I checked in the postgis container, the table users with psql:

ghc=# SELECT * FROM user;
 user 
------
 ghc
(1 row)

How do I log in? :(

epifanio
@epifanio
where can I change admin password in the DB ?
Just van den Broecke
@justb4
@epifanio GHC stores encrypted passwords in the user table. Even as an admin one cannot decrypt user passwords. What it looks like is that SELECT * FROM user shows the Postgres User table. Looks like you are in the Postgres DB. The GHC user table is in public.user but you must use psql to login into that DB. First make sure you have initialized a GHC DB in Postgres. See also the documentation on GHC password management.
epifanio
@epifanio
@justb4 thanks! logging in and check if I can update the admin password
the testing endpoint is at https://ghck.epinux.com
Just van den Broecke
@justb4
@epifanio Looks good, registered endpoints! Yes you can update the password as in doc above. Or reset via email if configured ok.
epifanio
@epifanio
yes, still playing with the config. I see it has 'locaalhost' hardcoded in the mail message .. looking into it now
Just van den Broecke
@justb4
You must configure an SMTP endpoint for that, also to get notifications by email. Some use a gmail account (be sure to allow "untrusted" apps in gmail config).
epifanio
@epifanio
yes, I already have it set up this way, using an app-passwd from gmail.
the email that I receive is showing the correct value for the test, but has the localhost variable in it - I guess it comes from the
GHC_SITE_URL=''
value from the dockerfile. I am now overriding it in the ghc.env
epifanio
@epifanio
the relevant part of the log: (Background on this error at: http://sqlalche.me/e/e3q8)
epifanio
@epifanio
epifanio
@epifanio
I can't find in the code where the db connection is creatred (usually 'create_engine' in sqlalchemy) but seems such call is hidden behind flask_sqlalchemy
epifanio
@epifanio
How to tell the flask
_sqlalchemy to read the following option?
    SQLALCHEMY_ENGINE_OPTIONS = {
                                 'pre_ping': True
                                 }
Just van den Broecke
@justb4
From what I see in the log there is a DB failure on the SELECT on Resource table, never any other query like the ResourceLock table:
SELECT resource.auth AS resource_auth, resource.identifier AS resource_identifier, resource.resource_type AS resource_resource_type, resource.active AS resource_active, resource.title AS resource_title, resource.url AS resource_url, resource.latitude AS resource_latitude, resource.longitude AS resource_longitude, resource.owner_identifier AS resource_owner_identifier, resource.run_frequency AS resource_run_frequency FROM resource
I never see these errors on demo.geohealthcheck.org, which runs the latest Docker Image. Are you running with Docker and the latest version?
Yes, GHC_SITE_URL determines the, URL mentioned in emails.
epifanio
@epifanio
Yes, I am running it with docker - using the geopython docker image (latest built)
mysettings is to run GHCK in a docker swarm proxied with traefik
we experencied a similar issue with pycsw with the same settings
it went away using this approach geopython/pycsw#696
epifanio
@epifanio
you can test it by loading the service at this url: https://ghck.epinux.com/ i will leave it untouched for a while, the internal server error happes when the app is cold nooane using it ..
Just van den Broecke
@justb4
Interesting, loosing DB-connections (and reconnecting) is often an issue with databases. The Docker startup script does something familiar in the configure.sh.
It looks like these can be configured in Flask_SQLAlchemy, pssobly in the standard GHC config file. But how to pass these to the Docker Containers is a bit a puzzle, also as SQLALCHEMY_ENGINE_OPTIONS is a dict. But as more and more deploys will use cloud technology like Kubernetes and Swarm we must solve this. PR welcome!
Just van den Broecke
@justb4
In init.py the SQLAlchemy object is created which has an engine_options parameter.
epifanio
@epifanio
i will start adding the pre-ping as option to the engine and see how it behavies
Just van den Broecke
@justb4
Ok, great!
epifanio
@epifanio
testingthe changes on https://ghck.epinux.com/ it seems there is no longer internal server error.
epifanio
@epifanio
@justb4 the PR should fix the internal server error I was having. One qestion: I first started the envirnment without setting the notification email addresses -- so it was using the default value of: admin.istrator@mydomain.com
I then restarted the service using a configured environment with a proper list of email that will receive the notification.
I noticed that the old email address: admin.istrator@mydomain.com is still used by the system. I guess its entry didn't get removed from the db
Just van den Broecke
@justb4
Thanks @epifanio for PR #400. I added my review. Main question is: how much overhead adds pre_ping, as the engine/dialect will issue a SELECT 1 for each and every DB-access, at least when accessing a DB-connection. Discussion in PR. But the feature is very useful, e.g. also to survive DB restarts.
epifanio
@epifanio
Thanks for you comment @justb4 I added a reply
I agree having it as a configuarble makes things more flexible
i can amend the PR according with the comment
epifanio
@epifanio
@justb4 how does it look geopython/GeoHealthCheck#400 ?
Just van den Broecke
@justb4
@epifanio see my review there.
epifanio
@epifanio
@justb4 great! thanks, I will work on it :)