I have a bunch of data in splunk but it’s not really HDRHistogram format, if I can extract time when a request came in and number of miliseconds it took to fulfill, is that enough to pipe into something that will give me HDRHistogram output?
I don't know of an existing tool for that but it should be straightforward to write a little tool that does that.
Are you looking for something that updates live or just a batch process to run after the fact?
batch process is enough
I will probably export from splunk and then process after that
OK. In that case, I would simply write a main method that reads, say, an integer per line from stdin and shoves it in a HdrHistogram, then writes to an output file of your choosing once stdin is closed.
is it not important when did the read happen?
Nope. You basically have a sequence of measurements (or, if you like, infinitely dense measurements, a finite amount of which are nonzero) of service time or what have you. The important thing is to capture all the different nonzero elements, and to not throw away outliers.
ok
Certainly there may be things where you care about the ordering, etc, but that's not what HdrH is for. You lose ordering for the sake of compact representation.
that’s easy
It's also not inconceivable that you will wish to bucket your histograms into, say, 1 or 5 or 10 minute chunks of data
the window
since you can easily add the smaller HdrH's together to form a "all day" one, for instance
sure
@schrepfler: Actually, jHiccup can do this. It's got a [-f inputFileName] option that will take a file formatted with two column (timestamp and latency) and produce a histogram log from that. See README under "Using jHiccup to process latency log files"
I built this mode in because I often get data that can be converted to that form (response time data from logs, jMeter logs, GC logs, etc.) and want to view them by percentile distribution and by behavior over time.
awesome, thanks gil!
I watched your talk posted on infoq on how NOT to measure latency and that got me intrested
we’ve got a spingmvc app and we log precisely the miliseconds per request
so I wanted to see what will come out
There is even a "secret" fill-zeros mode (-fz) that is very useful for plotting GC logs. It takes the latency input as reports of pauses, presuming that the rest of the time range is 0s. It then produces a histogram log for how long a 0-length operation would take in the JVM if that GC log represented it's pauses.
Sort of like a jHiccup equivalent, but purely by post-processing the GC logs.
@marshallpierce: Sorry for not getting back to the previous conversation. I've been bogged down with stuff. Traveling again today, so I'll have some time to catch up on the plane.
@giltene no problem. I've got plenty of other parts of the impl to port to keep me busy.
Phaser will be interesting to squeeze into Rust's memory model
the finagle guys are adding an hdrhistogram implementation to the next twitter server as well, it’s not aiming for perfection but with 0.5% precision with 7kb per stat
@schrepfler by the way, it should be straightforward to record service time directly in a springmvc interceptor if you so choose
I'm not sure if there is already an existing springmvc -> metrics integration, but if there is, you can get those measurements to a HdrHistogram with https://bitbucket.org/marshallpierce/hdrhistogram-metrics-reservoir
we have that
as miliseconds to respond to the request
ah, I didn’t get this second part, thanks gitter
@marshallpierce : Have you looked at pushing your metrics work upstream?
yeah, we hired the maintainer of metrics (Ryan Tenney) to my last company, which I then promptly left (for unrelated reasons), and he expressed interest in merging in my reservoir impl
but he knows it's on my radar, along with rewriting the reporting pipeline to be 0-alloc instead of mega-alloc as it currently is
My new job is much more open source friendly so I should have nontrivial time to dedicate to metrics.
@marshallpierce : Ah, didn't know you moved to TrueVault
Getting Metrics (the upstream version) to use HdrHistigram would be a great step towards pressuring time-series-databases to accept it as a natural value type.
Then we need to make a StatsD version that knows how to carry it.
influxdb still doesn’t have support for it correct?
I think one reason it's not natural for TSDBs to support it is that there aren't a lot of "data sources" that produce it. And the other way around, too (a chicken and egg problem).
The fact that we have a wire format should help, and the multiple implementations (Java/C/C#/Pythin/Go/Erlang/Rust) are a good hint that it's needed.
I think it's also a problem is that people don't concern themselves with statistical validity. They have a number, and it looks like any other number, so you should be able to add / multiply / divide
@marshallpierce : what's the challenge with the Rust memory model? Is it that it's not actually defined?
It's more that I don't know enough Rust. I feel clumsy coming from Java that I know so well
Rust has overall been a delight, but learning how to use the borrow checker to my advantage has been the sort of "advanced" thing that usually doesn't come so early in a language's learning curve.
Anyway, I'm slowly formulating some thoughts on how to view all metrics as time-domain functions, and the sort of measurements that we can take on them as different forms of compression. In somewhat sloppy terms, viewing everything through a nonparametric lens (we don't know what the distributions are, so where can we go from there)
Looks like there is no formal Rust memory model (yet); there are zillion-post-long github issues hashing it out
If things start with lossless recording and go on from there with filters and compression schemes that have agreed-upon behaviors (e.g. regarding loss of precision in time and in magnitude), then this time-domain approach would be good.
I think things get hard when multiple systems get involved in handing these things around. E.g. app->Metrics->StatsD->Graphite is a common path for data to take.
And there are usually summarizations going on at these handoffs.
Right. I think the only way to get a correct (or at least bounded loss of precision) pipeline is to have some math under the hood
E.g. App->Metrics and/or Metircs->StatsD and/or StatsD->Graphite will often summarize from a lossless (record every metric) to a lossy (compressed in some form losing time and/or value accuracy) representation
HdrHistogram intervals represent such a lossy step, with defined accuracy loss (don't know when within the intervals, and +/- magnitude precision specified for the histogram).
[In fact, any histogram will represent a similar thing, HdrHistogram is just a good example]
By keeping the values in a histogram representation, they also remain in a domain where aggregation math is valid. E.g. you can add histograms across intervals and still have a valid histogram of the magnitude behavior across the cumulative interval. Or you can add histograms from many parts of an overall system (e.g. many servers in a cluster) to get a valid histogram across the whole system.
It's this aggregation step that is unfortunately not valid for summarized data that TSDBs currently tend to carry (e.g. percentile values, or even averages).
Indeed. That's what I'm trying to figure out a way to express: starting from time domain, we assume that the data volume is too high to communicate directly, so we aggregate into windows of whatever granularity makes sense, and then define what operations make sense from there (e.g. a counter is monotonically increasing, so we can use different compression than for service time)
If we could get the app-Metrics, Metrics->StatsD, StatsD->Graphite, and Graphite->underlyingTSDB steps to all use histograms as their value types (with HdrHistogram being a good representation of one), we could change this...
@schrepfler : BTW if you haven't played with it before, you amy wan to look at HdrHistogramVisualizer . It's a bit crude, but a great way to look at histogram logs to view them in both the percentile distribution and time domains at the same time.
We really need a Javascript port so that we can do this in the browser like we currently do [only] for percentiles at http://hdrhistogram.github.io/HdrHistogram/plotFiles.html
thanks Gil
if you get a JS port you could have it running on node as well
when I use jHiccupLogProcessor, does the timestamp have to be unix timestamp or other formats are acceptable?
The jHiccupLogProcessor and HistogramLogProcessor are the same thing (I included the jHiccup one for people who use jHiccup and don't want to know about HdrHistogram). The timestamps in the log can be based off an arbitrary base time. You can see the logic for how it is parsed by following the logic that computes absoluteStartTimeStampSec here: https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogReader.java#L240
The annoying thing about Javascript is that there are no 64-bit integers.
Doesn't webGL cover support for typed arrays of bytes and other numeric types? That might work
Might do, it has a GLint64 type that can be used in a certain context, but not sure what the Node support for that would be like.
If you ask me, not being able to use good number types is one of the penalties you pay for choosing Node ;)
My thinking for a JavaScript port is to start with counts lik
Limited to 2^53
As long as you don't overflow that, js can deal with it just fine. If it does overflow, idk yet.
Maybe accept loss of precision in counts
The encoding/decoding from wire format will be a bit annoying, but not a huge deal
encoding/decoding is probably okay, there's some buffer classes that can be useful.