These are chat archives for HdrHistogram/HdrHistogram

10th
Apr 2016
Srepfler Srdan
@schrepfler
Apr 10 2016 15:34
I have a bunch of data in splunk but it’s not really HDRHistogram format, if I can extract time when a request came in and number of miliseconds it took to fulfill, is that enough to pipe into something that will give me HDRHistogram output?
Marshall Pierce
@marshallpierce
Apr 10 2016 15:36
I don't know of an existing tool for that but it should be straightforward to write a little tool that does that.
Are you looking for something that updates live or just a batch process to run after the fact?
Srepfler Srdan
@schrepfler
Apr 10 2016 15:37
batch process is enough
I will probably export from splunk and then process after that
Marshall Pierce
@marshallpierce
Apr 10 2016 15:38
OK. In that case, I would simply write a main method that reads, say, an integer per line from stdin and shoves it in a HdrHistogram, then writes to an output file of your choosing once stdin is closed.
Srepfler Srdan
@schrepfler
Apr 10 2016 15:39
is it not important when did the read happen?
Marshall Pierce
@marshallpierce
Apr 10 2016 15:41
Nope. You basically have a sequence of measurements (or, if you like, infinitely dense measurements, a finite amount of which are nonzero) of service time or what have you. The important thing is to capture all the different nonzero elements, and to not throw away outliers.
Srepfler Srdan
@schrepfler
Apr 10 2016 15:41
ok
Marshall Pierce
@marshallpierce
Apr 10 2016 15:41
Certainly there may be things where you care about the ordering, etc, but that's not what HdrH is for. You lose ordering for the sake of compact representation.
Srepfler Srdan
@schrepfler
Apr 10 2016 15:41
that’s easy
Marshall Pierce
@marshallpierce
Apr 10 2016 15:42
It's also not inconceivable that you will wish to bucket your histograms into, say, 1 or 5 or 10 minute chunks of data
Srepfler Srdan
@schrepfler
Apr 10 2016 15:42
the window
Marshall Pierce
@marshallpierce
Apr 10 2016 15:42
since you can easily add the smaller HdrH's together to form a "all day" one, for instance
Srepfler Srdan
@schrepfler
Apr 10 2016 15:42
sure
Gil Tene
@giltene
Apr 10 2016 15:47
@schrepfler: Actually, jHiccup can do this. It's got a [-f inputFileName] option that will take a file formatted with two column (timestamp and latency) and produce a histogram log from that. See README under "Using jHiccup to process latency log files"
I built this mode in because I often get data that can be converted to that form (response time data from logs, jMeter logs, GC logs, etc.) and want to view them by percentile distribution and by behavior over time.
Srepfler Srdan
@schrepfler
Apr 10 2016 15:50
awesome, thanks gil!
I watched your talk posted on infoq on how NOT to measure latency and that got me intrested
we’ve got a spingmvc app and we log precisely the miliseconds per request
so I wanted to see what will come out
Gil Tene
@giltene
Apr 10 2016 15:52
There is even a "secret" fill-zeros mode (-fz) that is very useful for plotting GC logs. It takes the latency input as reports of pauses, presuming that the rest of the time range is 0s. It then produces a histogram log for how long a 0-length operation would take in the JVM if that GC log represented it's pauses.
Sort of like a jHiccup equivalent, but purely by post-processing the GC logs.
@marshallpierce: Sorry for not getting back to the previous conversation. I've been bogged down with stuff. Traveling again today, so I'll have some time to catch up on the plane.
Marshall Pierce
@marshallpierce
Apr 10 2016 15:54
@giltene no problem. I've got plenty of other parts of the impl to port to keep me busy.
Phaser will be interesting to squeeze into Rust's memory model
Srepfler Srdan
@schrepfler
Apr 10 2016 15:56
the finagle guys are adding an hdrhistogram implementation to the next twitter server as well, it’s not aiming for perfection but with 0.5% precision with 7kb per stat
Marshall Pierce
@marshallpierce
Apr 10 2016 15:58
@schrepfler by the way, it should be straightforward to record service time directly in a springmvc interceptor if you so choose
I'm not sure if there is already an existing springmvc -> metrics integration, but if there is, you can get those measurements to a HdrHistogram with https://bitbucket.org/marshallpierce/hdrhistogram-metrics-reservoir
Srepfler Srdan
@schrepfler
Apr 10 2016 16:02
we have that
as miliseconds to respond to the request
ah, I didn’t get this second part, thanks gitter
Gil Tene
@giltene
Apr 10 2016 16:27
@marshallpierce : Have you looked at pushing your metrics work upstream?
Marshall Pierce
@marshallpierce
Apr 10 2016 16:28
yeah, we hired the maintainer of metrics (Ryan Tenney) to my last company, which I then promptly left (for unrelated reasons), and he expressed interest in merging in my reservoir impl
but he knows it's on my radar, along with rewriting the reporting pipeline to be 0-alloc instead of mega-alloc as it currently is
My new job is much more open source friendly so I should have nontrivial time to dedicate to metrics.
Gil Tene
@giltene
Apr 10 2016 16:39
@marshallpierce : Ah, didn't know you moved to TrueVault
Getting Metrics (the upstream version) to use HdrHistigram would be a great step towards pressuring time-series-databases to accept it as a natural value type.
Then we need to make a StatsD version that knows how to carry it.
Srepfler Srdan
@schrepfler
Apr 10 2016 16:41
influxdb still doesn’t have support for it correct?
Gil Tene
@giltene
Apr 10 2016 16:42
I think one reason it's not natural for TSDBs to support it is that there aren't a lot of "data sources" that produce it. And the other way around, too (a chicken and egg problem).
The fact that we have a wire format should help, and the multiple implementations (Java/C/C#/Pythin/Go/Erlang/Rust) are a good hint that it's needed.
Marshall Pierce
@marshallpierce
Apr 10 2016 16:44
I think it's also a problem is that people don't concern themselves with statistical validity. They have a number, and it looks like any other number, so you should be able to add / multiply / divide
Gil Tene
@giltene
Apr 10 2016 16:44
@marshallpierce : what's the challenge with the Rust memory model? Is it that it's not actually defined?
Marshall Pierce
@marshallpierce
Apr 10 2016 16:45
It's more that I don't know enough Rust. I feel clumsy coming from Java that I know so well
Rust has overall been a delight, but learning how to use the borrow checker to my advantage has been the sort of "advanced" thing that usually doesn't come so early in a language's learning curve.
Anyway, I'm slowly formulating some thoughts on how to view all metrics as time-domain functions, and the sort of measurements that we can take on them as different forms of compression. In somewhat sloppy terms, viewing everything through a nonparametric lens (we don't know what the distributions are, so where can we go from there)
Looks like there is no formal Rust memory model (yet); there are zillion-post-long github issues hashing it out
Gil Tene
@giltene
Apr 10 2016 16:50
If things start with lossless recording and go on from there with filters and compression schemes that have agreed-upon behaviors (e.g. regarding loss of precision in time and in magnitude), then this time-domain approach would be good.
I think things get hard when multiple systems get involved in handing these things around. E.g. app->Metrics->StatsD->Graphite is a common path for data to take.
And there are usually summarizations going on at these handoffs.
Marshall Pierce
@marshallpierce
Apr 10 2016 16:51
Right. I think the only way to get a correct (or at least bounded loss of precision) pipeline is to have some math under the hood
Gil Tene
@giltene
Apr 10 2016 16:52
E.g. App->Metrics and/or Metircs->StatsD and/or StatsD->Graphite will often summarize from a lossless (record every metric) to a lossy (compressed in some form losing time and/or value accuracy) representation
HdrHistogram intervals represent such a lossy step, with defined accuracy loss (don't know when within the intervals, and +/- magnitude precision specified for the histogram).
[In fact, any histogram will represent a similar thing, HdrHistogram is just a good example]
By keeping the values in a histogram representation, they also remain in a domain where aggregation math is valid. E.g. you can add histograms across intervals and still have a valid histogram of the magnitude behavior across the cumulative interval. Or you can add histograms from many parts of an overall system (e.g. many servers in a cluster) to get a valid histogram across the whole system.
It's this aggregation step that is unfortunately not valid for summarized data that TSDBs currently tend to carry (e.g. percentile values, or even averages).
Marshall Pierce
@marshallpierce
Apr 10 2016 16:57
Indeed. That's what I'm trying to figure out a way to express: starting from time domain, we assume that the data volume is too high to communicate directly, so we aggregate into windows of whatever granularity makes sense, and then define what operations make sense from there (e.g. a counter is monotonically increasing, so we can use different compression than for service time)
Gil Tene
@giltene
Apr 10 2016 16:58
If we could get the app-Metrics, Metrics->StatsD, StatsD->Graphite, and Graphite->underlyingTSDB steps to all use histograms as their value types (with HdrHistogram being a good representation of one), we could change this...
@schrepfler : BTW if you haven't played with it before, you amy wan to look at HdrHistogramVisualizer . It's a bit crude, but a great way to look at histogram logs to view them in both the percentile distribution and time domains at the same time.
We really need a Javascript port so that we can do this in the browser like we currently do [only] for percentiles at http://hdrhistogram.github.io/HdrHistogram/plotFiles.html
Srepfler Srdan
@schrepfler
Apr 10 2016 18:25
thanks Gil
if you get a JS port you could have it running on node as well
Srepfler Srdan
@schrepfler
Apr 10 2016 18:40
when I use jHiccupLogProcessor, does the timestamp have to be unix timestamp or other formats are acceptable?
Gil Tene
@giltene
Apr 10 2016 19:50
The jHiccupLogProcessor and HistogramLogProcessor are the same thing (I included the jHiccup one for people who use jHiccup and don't want to know about HdrHistogram). The timestamps in the log can be based off an arbitrary base time. You can see the logic for how it is parsed by following the logic that computes absoluteStartTimeStampSec here: https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogReader.java#L240
Michael Barker
@mikeb01
Apr 10 2016 21:14
The annoying thing about Javascript is that there are no 64-bit integers.
Marshall Pierce
@marshallpierce
Apr 10 2016 21:21
Doesn't webGL cover support for typed arrays of bytes and other numeric types? That might work
Michael Barker
@mikeb01
Apr 10 2016 21:25
Might do, it has a GLint64 type that can be used in a certain context, but not sure what the Node support for that would be like.
Marshall Pierce
@marshallpierce
Apr 10 2016 21:30
If you ask me, not being able to use good number types is one of the penalties you pay for choosing Node ;)
Gil Tene
@giltene
Apr 10 2016 22:20
My thinking for a JavaScript port is to start with counts lik
Limited to 2^53
As long as you don't overflow that, js can deal with it just fine. If it does overflow, idk yet.
Maybe accept loss of precision in counts
Gil Tene
@giltene
Apr 10 2016 22:26
The encoding/decoding from wire format will be a bit annoying, but not a huge deal
Michael Barker
@mikeb01
Apr 10 2016 23:14
encoding/decoding is probably okay, there's some buffer classes that can be useful.