Nothing as good as Gorilla though. That would complicate ingestion.
Hi again, can i query the raw values within a range using heroic? I am not sure if the aggregation (https://spotify.github.io/heroic/#!/docs/api/post-query-metrics) is optional.
Hello @udoprog ! I was wondering if you had a chance to look at spotify/heroic#371. Build is failing coverage, and I was wondering if you have any pointers about how to solve this. I actually added an integration test (which is passing) . I suspect the codecov tests might be broken since they are pointing to stuff that was not changed on the PR that I submitted. E.g. https://codecov.io/gh/spotify/heroic/pull/371/changes
ERR: ShardError(nodes=[LocalClusterNode(localMetadata=NodeMetadata(version=0, id=0336e36b-8d5c-49bb-9d8d-b435eb9eaf32, tags={}, service=ServiceInfo(name=The Heroic Time Series Database, version=0.0.1-SNAPSHOT, id=heroic)))], shard={}, error=Some fetches failed (4) or were cancelled (0), caused by Some fetches failed (4) or were cancelled (0))
I'm attempting to update a rather ancient heroic deployment to 0.8.7 and from the kafka consumer to the
+pubsub consumer, and am running into a frankly bizarre issue: messages with a certain key format appear to be silently not recorded in the metrics backend.
+pubsub consumer, and am running into a frankly bizarre issue: messages with a certain key format appear to be silently not recorded in the metrics backend.
I've turned on debug logging and the pubsub consumer appears to be correctly ingesting messages:
15:39:20.168 [Gax-15] DEBUG com.spotify.heroic.consumer.pubsub.Receiver - Received ID:495196398495706 with content:
{"uuid":"ff4bf904-41fe-43d8-bede-3212359aa7f8:60","timestamp":1554478740000,"value":130.0,"metric":"test130","aggregation":"max"}
heroic correctly updates the metadata store for that key:
$ curl -s -H "Content-Type: application/json" http://heroic:8080/metadata/series \
-d '{"range": {"type": "relative", "unit": "HOURS", "value": 1}, "filter": ["key", "ff4bf904-41fe-43d8-bede-3212359aa7f8:60"]}' \
| jq '.series[0]'
{
"key": "ff4bf904-41fe-43d8-bede-3212359aa7f8:60",
"tags": {
"aggregation": "deltasum",
"metric": "test136"
},
"resource": {}
}
...but it doesn't seem to actually be storing the data:
$ curl -s -H "Content-Type: application/json" http://heroic:8080/query/metrics \
-d '{"range": {"type": "relative", "unit": "HOURS", "value": 1}, "filter": ["key", "ff4bf904-41fe-43d8-bede-3212359aa7f8:60"]}' \
| jq '.result'
[]
the data just gets dropped on the floor
and just to make it exceedingly weird: this only happens when the key name ends in
:60 or :600 (these keys are generated by a rollup process we run as a beam job)
if the key name doesn't end like that, the data is correctly stored:
$ curl -s -H "Content-Type: application/json" http://heroic:8080/query/metrics \
-d '{"range": {"type": "relative", "unit": "HOURS", "value": 1}, "filter": ["key", "ff4bf904-41fe-43d8-bede-3212359aa7f8"]}' \
| jq '.result[0].values[0:2]'
[
[
1554477496000,
115
],
[
1554477497000,
115
]
]
:confounded:
this is heroic release 0.8.7 running on openjdk8 with the bigtable metrics backend and elasticsearch metadata backend, although I was able to reproduce the same behavior with the in-memory metadata and metrics stores.