Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Jakob Miksch
@JakobMiksch
I can run GHC on the RaspberryPi using docker-compose. However, there is still an error with ghc_web. But I assume it is not related to the RaspberryPi:
Traceback (most recent call last):                                                                    
  File "/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker             
    worker.init_process()                                                                             
  File "/venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 133, in init_process   
    self.patch()                                                                                      
  File "/venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 124, in patch          
    eventlet.monkey_patch()                                                                           
  File "/venv/lib/python3.7/site-packages/eventlet/patcher.py", line 296, in monkey_patch             
    from eventlet.support import psycopg2_patcher                                                     
  File "/venv/lib/python3.7/site-packages/eventlet/support/psycopg2_patcher.py", line 27, in <module> 
    import psycopg2                                                                                   
  File "/venv/lib/python3.7/site-packages/psycopg2/__init__.py", line 68, in <module>                 
    from psycopg2 import extensions as _ext                                                           
  File "/venv/lib/python3.7/site-packages/psycopg2/extensions.py", line 211, in <module>              
    from psycopg2. _range import Range                              # noqa                            
  File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 502, in <module>                  
    oid=3904, subtype_oid=23, array_oid=3905)                                                         
  File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 283, in __init__                  
    self._create_ranges(pgrange, pyrange)                                                             
  File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 302, in _create_ranges            
    if isinstance(pgrange, basestring):                                                               
NameError: name 'basestring' is not defined
Jakob Miksch
@JakobMiksch
I get the same error on my desktop computer. I opened an issue: geopython/GeoHealthCheck#412
Just van den Broecke
@justb4
Ok, see my comment there.
@nagyrobir @epifanio there was an NO(rwegian) translation file somewhere? Could not find back.
If working with Git/PRs is a problem, you can open an issue and attach the .po file there.
@justb4 This should be the link to the file https://files.gitter.im/575f1fdcc2f0db084a1db0a6/JEQ5/NO.po
Just van den Broecke
@justb4
Ok, have at least opened the issue #414. Need to find time, but ideal first issue for newcomers.
That is how I started joining GHC dev (NL translations)!
Just van den Broecke
@justb4
Fixed annoying issue #418. CI/CD now successful again, was due to stale ESRI FeatureServer URL used in test fixtures.
Astrid Emde
@astroidex
Hi all. Hope you are fine.
I have a question. Many of my WMS in GHC sho in the status section "No runs yet". I defined that they should run every 1440 min. Do you have an idea why they are not run. Also looks like the Services with "No runs yet" status get more and more.
Just van den Broecke
@justb4
@astroidex hope you as well. First make sure that each WMS Resource has the status Activ(e) True. That is the default.
Every 1440 min is every 24 hour. What could be, if you just created the WMS Resources in GHC, that they are still to be run.
The Scheduler sets a random time for the first Run within the time delta of the frequency , as to prevent that many Probes run at the same time, e.g. after a restart. There were also some changes there recently like PR #385. But overall this could mean that when scheduled every 24h, it could take up to 24h for the first run.
Usually Resources run in terms of minutes so this effect is not visible. We may need to adapt the schedule randomizer.
Etienne Pelletier
@Dukestep
@justb4, similarly to the question asked by Astrid, we have a resource that is schedule to run every 24 hrs to check the status code of a URL. That status code is updated depending on a file that is updated once daily. Currently, the random timing of the check makes it so that the check occurs a couple of hours before the change, and therefore we must wait a significant portion of the day before GHC notifies us again. Is there a possibility to run the check at a specific time via the web app without having to resort to the cron scheduling I see mentioned in the documentation?
Just van den Broecke
@justb4
@astroidex : did my answer clarify your situation?
@Dukestep : GHC uses an internal Python-based scheduler. There is no functionality to schedule at a specific time. The random timing was introduced to solve problems, a bit similar to "soldiers marching in-step on a bridge". This especially applied to smaller time-delta's. But 24h schedules could suffer from the current solution. Maybe for those cases the scheduling algorithm could be refined. Note that the randomness is (out of my head) only for the first run. From then the interval is constant.
Scheduling at exactly a specific time would be a new feature with some impact, like database (now only "frequency" in mins attribute).
Just van den Broecke
@justb4
#409 was just merged. Thanks to @jochemthart1 Vector Tiles Probe "MapBox TileJSON", got major fixes. Just tested three different implementations on GHC demo server that were all failing before, but are now succeeding. In several cases now a lat/lon parameter needs to be provided. Also T-rex working, heads up for @pka.
Astrid Emde
@astroidex
@justb4 Hi @justb4, thanks for your answer. Feb 07
After a restart all services are checked regularly. So it works fine again. Thanks for your answer and sorry for the late reply.
Just van den Broecke
@justb4
/all our GeoHealthCheck talk for FOSS4G 2022 was accepted! Schedule will follow.
Tom Kralidis
@tomkralidis
:+1:
Tom Kralidis
@tomkralidis

Hi all: in preparation for this year’s OSGeo Annual General Meeting (AGM), it would be great to have an update on GHC. We can make updates in:

https://docs.google.com/presentation/d/1Mis580FlD29HjaymfL9OC72X9IqcM4uvnqVH57jQLqI/edit#slide=id.g9401e4b2da_178_0

Thanks in advance (cc @justb4)

Just van den Broecke
@justb4
Dear GHC users/developers. The GHC-team is currently in the process of drafting a new GeoHealthCheck release: 0.9.0, as follow-up from 0.8.3.
More on this later today!
Just van den Broecke
@justb4
.... and we are happy to announce the release of GeoHealthCheck 0.9.0.
For more specifics, zie the upgrade notes or directly the Milestone issues and PRs that went in. Thanks for your patience and to all who contributed!
Just van den Broecke
@justb4
was also a good moment to add Release Management procedure:
https://github.com/geopython/GeoHealthCheck/wiki/ReleaseManagement
Tom Kralidis
@tomkralidis
Awesome work all!
Just van den Broecke
@justb4

Hi all: in preparation for this year’s OSGeo Annual General Meeting (AGM), it would be great to have an update on GHC. We can make updates in:

https://docs.google.com/presentation/d/1Mis580FlD29HjaymfL9OC72X9IqcM4uvnqVH57jQLqI/edit#slide=id.g9401e4b2da_178_0

Thanks in advance (cc @justb4)

done

Massimo Di Stefano
@epifanio
Hello GHC :) back at work after FOSS4G, it was great to finally meet with some of you!
Just van den Broecke
@justb4
Yes, good to have met you (again) too @epifanio !
Just van den Broecke
@justb4
Interesting issue #436. My idea afterwards:
Sometimes we want to change the admin user/password. There is a scenario in the docs, but a bit cumbersome.
I think the "admin password reset" could be performed more conveniently, independent from using Docker or a particular DB. Possibly via updating the config params or maybe a CLI though that has some security implications. What is currently already possible if the email settings are properly configured, to do a password reset via "lost password".
Charlie Negri
@charlienegri
hi, is this a good place to ask something about configuring a webhook from ghc or is there a more appropriate room?
Just van den Broecke
@justb4
@charlienegri yes, welcome, this is the right place!
Charlie Negri
@charlienegri
ok great, thanks! I am trying to configure a webhook from a ghc resource to PagerDuty
the PD endpoint that accepts post request is https://events.pagerduty.com/v2/enqueue
but the payload must contain some mandatory fields
so am trying to add some custom fields as indicated in the docs but none of what I tried works, I suspect it's because of how the webhook post request is set up in https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/notifications.py#L176
any reason why you have there r = requests.post(url, params) instead of for example r = requests.post(url, json=params)?
an example of a working post request to PagerDuty for me would be (in python)
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger"})
Charlie Negri
@charlienegri
and I cannot obtain something similar via the "Notify webhook" field in the ghc resource configuration
I tried a lot of variations from
https://events.pagerduty.com/v2/enqueue

{"routing_key":"[mykey]","event_action":"trigger","payload": {"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}}
to
https://events.pagerduty.com/v2/enqueue

routing_key=[mykey]
event_action=trigger
payload.summary=failing-ghc-probe
payload.source=my-csw-service
payload.severity=error
do you think I can obtain a successful post to the PD endpoint with the current ghc code? or do I have to fork and modify the request.post in do_webhook to fit my case?
Charlie Negri
@charlienegri
for reference, the accepted payload format for PagerDuty is here https://developer.pagerduty.com/api-reference/368ae3d938c9e-send-an-event-to-pager-duty
Just van den Broecke
@justb4
@charlienegri the webhook facility came from a contribution/PR. The current implementation may not cover all use-cases. I am not even sure how the data is formatted in the POST body. Looks like x-www-form-urlencoded (?). And that json= is mainly to specify params. You could offcourse write an interceptor that reformats/converts to PD format. I know there are cloud services to provide this.
But better is to fix within GHC, supporting multiple formats/content-types for the webhook POST.
Charlie Negri
@charlienegri
if I understand the code correctly there's no parsing currently, params is a bare dictionary that contains the ghc-specific fields (like ghc.result) plus the parsed extra arguments added in the Notify webhook form, so following the documentation of request.post, https://requests.readthedocs.io/en/latest/api/?highlight=POST#requests.post, that same dictionary could be passed with the nameddata= argument, and then it would be form-encoded, while the named argumentjson= encodes it specifically to JSON , see https://stackoverflow.com/a/47188297
for PagerDuty data= does not work, nor a bare dictionary, I need the latter, according to https://developer.pagerduty.com/api-reference/368ae3d938c9e-send-an-event-to-pager-duty and also consistently with the tests I made...
so, I suspect changing simply tojson=params in the requests.post function would work, but I haven't tested yet and idk if that fits the general needs of webhooks from ghc well enough..
are you aware of any working example of a currently working webhook setup? I have no idea what other endpoints may accept as payloads, generally speaking
Just van den Broecke
@justb4
@charlienegri yes you are right: the dict sent in the webhook POST is a merge of GHC-specific params and the ones in the notify-form. Then the webhook is sent as a POST-form-encoded. I think the original developer had a custom webapp that handled the incoming webhook. I don't know of specific examples, maybe from folks here on Gitter?
Looking at the PagerDuty API there are quite some requirements. To comply would be quite challenging, like a timestamp, URL to image etc. There are so many use cases, some may require AUth headers etc. One solution could be a GHC Webhook-Formatter Plugin.
But to have something working with the current implementation could be a forwarding proxy script. That could be as simple as a PHP script that handles and reformats incoming GHC Webhook requests. It could even be a Python Flask script. There are a lot of web integration services like https://ngrok.com/ that could help or even https://nodered.org/ .
Charlie Negri
@charlienegri
the only thing required to comply that I can't add is the json encoding tho, this request works for me:
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger"})

which means the following would work also

requests.post("https://events.pagerduty.com/v2/enqueue",json=params)

where params is the dictionary that results from parsing the input form + the ghc parameters, the output of _parse_webhook_location() basically

Charlie Negri
@charlienegri
the only difference between the two would be the added extra ghc params in the dictionary but they do not matter I think
yep, I tested that this also gets accepted:
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"csw-dev","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger","ghc.result":"result"})
Charlie Negri
@charlienegri
if having JSON encoding only is too restrictive, maybe an extra option in the form to choose between form encoding or json encoding could be a good strategy for you then?
as for my use-case I'll discuss with others how to proceed, probably editing the source code is the easiest for us (am not the person who set up the ghc instance)
Charlie Negri
@charlienegri
if you decide not to change anything, that is :)
Charlie Negri
@charlienegri
it's a cosmetic change for just get it working, r = requests.post(url, json=params) here https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/notifications.py#L220
the payload is not fancy, granted, but what matters to me is just to trigger an alert
Just van den Broecke
@justb4

Well, yes that would cover your use case. We have to deal with existing installations. I think the most generic solution is to specify in each webhook (there may be multiple): URL, POST Content-Type, Body Text (any, like JSON or XML) as Jinja2 Template that can integrate GHC Run params (now added as dict). That would be a longer-term solution. Would love to see a PR for this. We also need to upgrade existing installs DB-data (we do this automatically with Alembic). The GHC webhook Form would basically have 3 HTML Input widgets for adding/editing:

  • Text for the URL
  • Dropdown for Content-Type
  • TextArea for Body Template/Payload
    This would almost be like YARC or Postman form. I love to see a PR for this!

As an intermediate very minimal solution: the current webhook Form is an HTML TextArea which is interpreted/parsed as having 3 parts : a URL, an empty line, and the rest params. It also attempts to parse the latter as JSON. We could flag that as that application/json Content-Type is to be used, but this will break existing installations (that have filled-in JSON but expect Form-encoded webhook calls).
An option here is to have an optional Content-Type after the first URL-line, or an optional 4th part in the TextArea that specifies the Content-Type. If not present, the current default is used. Also happy to see a PR for this (or sponsored development).