"Accept: application/json", but maybe this is a direction with many interworking issues (over f=json).
Accept: headers via issue #313 .
User-Agent and compression Accept. Upgraded OWSLib to 0.20.0 (now WFS auth should work) and enhanced OGC OAFeat Probe support. German translation patches contributed by @astroidex.
GHC_RUNNER_IN_WEBAPP=True. The default for Docker is False. There has been some discussion and an issue around this: #303. We are a bit cautious to change this default as it may silently affect existing installs. Especially in a Docker deployment it is good practice to separate the 3 main GHC components (UI, Healthcheck Runner and database). docker-compose is a convenient way, see this example.
Check. You could write a Check Plugin for a specific 401-Check. Out of my head: the Run report would specify the specific HTTP-error code. I think both your options apply: AFAIK in HTTP (Basic, Token) Authentication is a one-time request/response: if there are no auth-headers or if the auth-headers contain the wrong credentials, a 401 response is returned. GHC supports two Auth methods: Basic and Bearer Token per Resource (endpoint) which can be configured in the UI and will be sent with each HTTP-request in the HTTP headers.
@MichaelOstling thanks, that is actually a scenario (add CSW and then monitor referenced OWS services) proposed at a FOSS4G presentation (think by GeoCat). This would require a new (CSW) Probe Plugin. This is similar to the GeoNode monitoring: a GeoNode endpoint is added and then all its referenced OWS-es endpoints are automatically fetched and added to a GHC instance. There is already a CSW Resource Type.
Export/Import JSON is really only used for Unit-testing, it is quite radical: wiping all existing data in the DB.
There is no API yet, it is our next-planned big Feature. Lack of time and funding. There is a spec already:
https://github.com/geopython/GeoHealthCheck/wiki/API-Specification .
@KoalaGeo yes, interesting question. Having some thoughts: there are many many FOSS monitoring frameworks, Prometheus/Grafana (I use mostly), Icinga, Datadog, etc. And Zabbix offcourse. True, a lot of what GHC is does is covered (scheduling, data collection, notification, visualization). The heart of GHC are the often extensive Probes/Checks for OGC and other APIs (like GeoNode). With a lot of complex work delegated to OWSLib. These would all have to be (re)written (for Zabbix Agent2) in GoLang. Plus the Probe/Check configuration-capabilities are extensive.
On the other hand integrations with existing monitoring systems has been projected for GHC at the level of data sampling for existing monitoring systems. The current GHC API already allows for summary data, so a simple core-Zabbix command-based shell extension could at least do a curl-type GHC API call and e.g. report number of failing Resources or even more. Even GHC Probes could be executed via commands, so that level of integration is also possible.
Hello Humans - I'm stucked starting runner bcs. pytz doesnt know Europe/UTC... any hints where to start troubleshooting?
pytz.exceptions.UnknownTimeZoneError: 'Europe/UTC'
1 reply
I added few records (CSW endpoints) via the web UI - but I was thinking to check if possible to update the db directly with a insert (so to add records in a batch)
- I also have to work on a thredds catalog probe (checking opendap resource url for a given xml catalog) - hope to be able to make a contribution.
Hi, I'm trying to run checks on 500 resources by adding them directly into the sqlite db through a script. However, some resources fail randomly (they might respond OK one time and error the next time). What I have noticed is that these failures happen more often when I set the run_frequency to a lower number and that these failures often happen at the same time. It is as if the failures are caused by the checks running too close to each other. Most failures happen when the checked_datetime is less than 0.1s after each other.
This might not give problems when adding resources through the interface, as these resources are added to the schedule at the moment they are added. However, when I add 500 resources to the database, they will be scheduled at whole minutes between now and now+run_frequency.
scheduler.py github line 247
This means that the chances are high that with 500 resources with a run_frequency of 60, there are on average ~10 checks each minute, which also start at approximately at the same time.
Something I found very interesting is that when I add a resource to those 500 through the interface, this resource never fails. The same is true for when I 'edit' a resource, it will not fail anymore as well. I still haven't been able to pinpoint what the difference is here.
For now I have GHC running with a change in scheduler.py where resources are scheduled with 5s in-between them instead of randomly determined. This works, but this means that it takes 45 minutes to complete for 500 resources and I would like to get it down to less than 10 minutes.
Do you happen to know why running resources so close to each other might give errors and if this is a common problem with checking many resources or if it is caused by the fact that they are added to the db through a script?
The scheduler phrase
timedelta(minutes=random.randint(0, freq)) is to ensure liveliness (like soldiers marching out of sync on a bridge).Would be interesting to know what the exact failures are and which version and environment (Docker?) of GHC you are using.
For one thing the GHC demo site demo.geohealthcheck.org runs stable for several years now though there are just 90 resources. The errored resources are usually in HTTP connectivity. That seems to come and go but usually the same resources. The latest version of GHC has improved on HTTP retry.
These kind of stress-tests are good to pop out concurrency errors e.g. in global data which is not supposed to be global as
scheduler.py runs a highly multithreaded environment. If there is really a problem, please open an issue and we'll work from there.
@justb4 Thanks for your quick answer.
I have tried both sqlite and PostgreSQL, but unfortunately that made no difference.
I am on version 0.8.3 and have tried both with and without docker.
The scheduler and webapp are running in seperate processes.
The error I get is most: Found no WMS layers or Found no TMS layers or Found no WFS features types. Sometimes a http 502 error.
With 500 resources, a run_frequency of 120 minutes has a failure rate of 0.1% for me so I can imagine that the demo with 90 resources per 60 minutes would have a very low chance of giving these errors. Btw, 500 resources on 60 minutes had a failure rate of around 7%.
I still can't wrap my head around the fact that resources added through the interface have never given me these same errors (yet). Maybe I should run a test adding a resource through interface and then clearing the run table in the DB and restarting GHC to see if they start giving errors then.
I will open an issue for it