I can run GHC on the RaspberryPi using docker-compose. However, there is still an error with ghc_web. But I assume it is not related to the RaspberryPi:
Traceback (most recent call last):
File "/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 133, in init_process
self.patch()
File "/venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 124, in patch
eventlet.monkey_patch()
File "/venv/lib/python3.7/site-packages/eventlet/patcher.py", line 296, in monkey_patch
from eventlet.support import psycopg2_patcher
File "/venv/lib/python3.7/site-packages/eventlet/support/psycopg2_patcher.py", line 27, in <module>
import psycopg2
File "/venv/lib/python3.7/site-packages/psycopg2/__init__.py", line 68, in <module>
from psycopg2 import extensions as _ext
File "/venv/lib/python3.7/site-packages/psycopg2/extensions.py", line 211, in <module>
from psycopg2. _range import Range # noqa
File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 502, in <module>
oid=3904, subtype_oid=23, array_oid=3905)
File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 283, in __init__
self._create_ranges(pgrange, pyrange)
File "/venv/lib/python3.7/site-packages/psycopg2/_range.py", line 302, in _create_ranges
if isinstance(pgrange, basestring):
NameError: name 'basestring' is not defined
@nagyrobir @epifanio there was an NO(rwegian) translation file somewhere? Could not find back.
If working with Git/PRs is a problem, you can open an issue and attach the .po file there.
If working with Git/PRs is a problem, you can open an issue and attach the .po file there.
@justb4 This should be the link to the file https://files.gitter.im/575f1fdcc2f0db084a1db0a6/JEQ5/NO.po
Ok, have at least opened the issue #414. Need to find time, but ideal first issue for newcomers.
That is how I started joining GHC dev (NL translations)!
That is how I started joining GHC dev (NL translations)!
Fixed annoying issue #418. CI/CD now successful again, was due to stale ESRI FeatureServer URL used in test fixtures.
@astroidex hope you as well. First make sure that each WMS Resource has the status
Every 1440 min is every 24 hour. What could be, if you just created the WMS Resources in GHC, that they are still to be run.
The Scheduler sets a random time for the first Run within the time delta of the frequency , as to prevent that many Probes run at the same time, e.g. after a restart. There were also some changes there recently like PR #385. But overall this could mean that when scheduled every 24h, it could take up to 24h for the first run.
Usually Resources run in terms of minutes so this effect is not visible. We may need to adapt the schedule randomizer.
Activ(e) True. That is the default.Every 1440 min is every 24 hour. What could be, if you just created the WMS Resources in GHC, that they are still to be run.
The Scheduler sets a random time for the first Run within the time delta of the frequency , as to prevent that many Probes run at the same time, e.g. after a restart. There were also some changes there recently like PR #385. But overall this could mean that when scheduled every 24h, it could take up to 24h for the first run.
Usually Resources run in terms of minutes so this effect is not visible. We may need to adapt the schedule randomizer.
@justb4, similarly to the question asked by Astrid, we have a resource that is schedule to run every 24 hrs to check the status code of a URL. That status code is updated depending on a file that is updated once daily. Currently, the random timing of the check makes it so that the check occurs a couple of hours before the change, and therefore we must wait a significant portion of the day before GHC notifies us again. Is there a possibility to run the check at a specific time via the web app without having to resort to the cron scheduling I see mentioned in the documentation?
@astroidex : did my answer clarify your situation?
@Dukestep : GHC uses an internal Python-based scheduler. There is no functionality to schedule at a specific time. The random timing was introduced to solve problems, a bit similar to "soldiers marching in-step on a bridge". This especially applied to smaller time-delta's. But 24h schedules could suffer from the current solution. Maybe for those cases the scheduling algorithm could be refined. Note that the randomness is (out of my head) only for the first run. From then the interval is constant.
Scheduling at exactly a specific time would be a new feature with some impact, like database (now only "frequency" in mins attribute).
@Dukestep : GHC uses an internal Python-based scheduler. There is no functionality to schedule at a specific time. The random timing was introduced to solve problems, a bit similar to "soldiers marching in-step on a bridge". This especially applied to smaller time-delta's. But 24h schedules could suffer from the current solution. Maybe for those cases the scheduling algorithm could be refined. Note that the randomness is (out of my head) only for the first run. From then the interval is constant.
Scheduling at exactly a specific time would be a new feature with some impact, like database (now only "frequency" in mins attribute).
#409 was just merged. Thanks to @jochemthart1 Vector Tiles Probe "MapBox TileJSON", got major fixes. Just tested three different implementations on GHC demo server that were all failing before, but are now succeeding. In several cases now a lat/lon parameter needs to be provided. Also T-rex working, heads up for @pka.
Hi all: in preparation for this year’s OSGeo Annual General Meeting (AGM), it would be great to have an update on GHC. We can make updates in:
Thanks in advance (cc @justb4)
.... and we are happy to announce the release of GeoHealthCheck 0.9.0.
For more specifics, zie the upgrade notes or directly the Milestone issues and PRs that went in. Thanks for your patience and to all who contributed!
For more specifics, zie the upgrade notes or directly the Milestone issues and PRs that went in. Thanks for your patience and to all who contributed!
was also a good moment to add Release Management procedure:
https://github.com/geopython/GeoHealthCheck/wiki/ReleaseManagement
https://github.com/geopython/GeoHealthCheck/wiki/ReleaseManagement
Hi all: in preparation for this year’s OSGeo Annual General Meeting (AGM), it would be great to have an update on GHC. We can make updates in:
Thanks in advance (cc @justb4)
done
Interesting issue #436. My idea afterwards:
Sometimes we want to change the admin user/password. There is a scenario in the docs, but a bit cumbersome.
I think the "admin password reset" could be performed more conveniently, independent from using Docker or a particular DB. Possibly via updating the config params or maybe a CLI though that has some security implications. What is currently already possible if the email settings are properly configured, to do a password reset via "lost password".
Sometimes we want to change the admin user/password. There is a scenario in the docs, but a bit cumbersome.
I think the "admin password reset" could be performed more conveniently, independent from using Docker or a particular DB. Possibly via updating the config params or maybe a CLI though that has some security implications. What is currently already possible if the email settings are properly configured, to do a password reset via "lost password".
the PD endpoint that accepts post request is https://events.pagerduty.com/v2/enqueue
but the payload must contain some mandatory fields
but the payload must contain some mandatory fields
so am trying to add some custom fields as indicated in the docs but none of what I tried works, I suspect it's because of how the webhook post request is set up in https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/notifications.py#L176
any reason why you have there
r = requests.post(url, params) instead of for example r = requests.post(url, json=params)?
an example of a working post request to PagerDuty for me would be (in python)
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger"})
I tried a lot of variations from
https://events.pagerduty.com/v2/enqueue
{"routing_key":"[mykey]","event_action":"trigger","payload": {"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}}
to
https://events.pagerduty.com/v2/enqueue
routing_key=[mykey]
event_action=trigger
payload.summary=failing-ghc-probe
payload.source=my-csw-service
payload.severity=error
do you think I can obtain a successful post to the PD endpoint with the current ghc code? or do I have to fork and modify the
request.post in do_webhook to fit my case?
for reference, the accepted payload format for PagerDuty is here https://developer.pagerduty.com/api-reference/368ae3d938c9e-send-an-event-to-pager-duty
@charlienegri the webhook facility came from a contribution/PR. The current implementation may not cover all use-cases. I am not even sure how the data is formatted in the POST body. Looks like
But better is to fix within GHC, supporting multiple formats/content-types for the webhook POST.
x-www-form-urlencoded (?). And that json= is mainly to specify params. You could offcourse write an interceptor that reformats/converts to PD format. I know there are cloud services to provide this.But better is to fix within GHC, supporting multiple formats/content-types for the webhook POST.
if I understand the code correctly there's no parsing currently,
for PagerDuty
so, I suspect changing simply to
are you aware of any working example of a currently working webhook setup? I have no idea what other endpoints may accept as payloads, generally speaking
params is a bare dictionary that contains the ghc-specific fields (like ghc.result) plus the parsed extra arguments added in the Notify webhook form, so following the documentation of request.post, https://requests.readthedocs.io/en/latest/api/?highlight=POST#requests.post, that same dictionary could be passed with the nameddata= argument, and then it would be form-encoded, while the named argumentjson= encodes it specifically to JSON , see https://stackoverflow.com/a/47188297for PagerDuty
data= does not work, nor a bare dictionary, I need the latter, according to https://developer.pagerduty.com/api-reference/368ae3d938c9e-send-an-event-to-pager-duty and also consistently with the tests I made...so, I suspect changing simply to
json=params in the requests.post function would work, but I haven't tested yet and idk if that fits the general needs of webhooks from ghc well enough..are you aware of any working example of a currently working webhook setup? I have no idea what other endpoints may accept as payloads, generally speaking
@charlienegri yes you are right: the dict sent in the webhook POST is a merge of GHC-specific params and the ones in the notify-form. Then the webhook is sent as a POST-form-encoded. I think the original developer had a custom webapp that handled the incoming webhook. I don't know of specific examples, maybe from folks here on Gitter?
Looking at the PagerDuty API there are quite some requirements. To comply would be quite challenging, like a timestamp, URL to image etc. There are so many use cases, some may require AUth headers etc. One solution could be a GHC Webhook-Formatter Plugin.
But to have something working with the current implementation could be a forwarding proxy script. That could be as simple as a PHP script that handles and reformats incoming GHC Webhook requests. It could even be a Python Flask script. There are a lot of web integration services like https://ngrok.com/ that could help or even https://nodered.org/ .
Looking at the PagerDuty API there are quite some requirements. To comply would be quite challenging, like a timestamp, URL to image etc. There are so many use cases, some may require AUth headers etc. One solution could be a GHC Webhook-Formatter Plugin.
But to have something working with the current implementation could be a forwarding proxy script. That could be as simple as a PHP script that handles and reformats incoming GHC Webhook requests. It could even be a Python Flask script. There are a lot of web integration services like https://ngrok.com/ that could help or even https://nodered.org/ .
the only thing required to comply that I can't add is the json encoding tho, this request works for me:
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"my-csw-service","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger"})which means the following would work also
requests.post("https://events.pagerduty.com/v2/enqueue",json=params)where params is the dictionary that results from parsing the input form + the ghc parameters, the output of _parse_webhook_location() basically
yep, I tested that this also gets accepted:
requests.post("https://events.pagerduty.com/v2/enqueue",json={"payload":{"summary":"failing-ghc-probe","source":"csw-dev","severity":"error"}, "routing_key":"[mykey]","event_action":"trigger","ghc.result":"result"})
as for my use-case I'll discuss with others how to proceed, probably editing the source code is the easiest for us (am not the person who set up the ghc instance)
it's a cosmetic change for just get it working,
the payload is not fancy, granted, but what matters to me is just to trigger an alert
r = requests.post(url, json=params) here https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/notifications.py#L220the payload is not fancy, granted, but what matters to me is just to trigger an alert
Well, yes that would cover your use case. We have to deal with existing installations. I think the most generic solution is to specify in each webhook (there may be multiple): URL, POST Content-Type, Body Text (any, like JSON or XML) as Jinja2 Template that can integrate GHC Run params (now added as dict). That would be a longer-term solution. Would love to see a PR for this. We also need to upgrade existing installs DB-data (we do this automatically with Alembic). The GHC webhook Form would basically have 3 HTML Input widgets for adding/editing:
- Text for the URL
- Dropdown for Content-Type
- TextArea for Body Template/Payload
This would almost be like YARC or Postman form. I love to see a PR for this!
As an intermediate very minimal solution: the current webhook Form is an HTML TextArea which is interpreted/parsed as having 3 parts : a URL, an empty line, and the rest params. It also attempts to parse the latter as JSON. We could flag that as that application/json Content-Type is to be used, but this will break existing installations (that have filled-in JSON but expect Form-encoded webhook calls).
An option here is to have an optional Content-Type after the first URL-line, or an optional 4th part in the TextArea that specifies the Content-Type. If not present, the current default is used. Also happy to see a PR for this (or sponsored development).