These are chat archives for HdrHistogram/HdrHistogram

29th
Sep 2015
Gil Tene
@giltene
Sep 29 2015 00:00
And @ahothan: REALLY Cool work on the python version. I've mentioned it in three talks this past week (one at Cassandra Summit and two at StrangeLoop). Thanks!
Now if we could only convince someone to do a javascript version....
Ivan Topolnjak
@ivantopo
Sep 29 2015 10:04
hey @giltene, I was experimenting with the idea of a Javascript version (Typescript, for sanity purposes :D) but a surprise I got was that in JS, operands of bitwise operations get converted to 32-bit integers so there would be a need for some not-so-fast operations to calculate which index to update. If we are talking latency, probably all measurements in a JS environment will be bellow the 32-bit limit given their time source resolution, but it felt a bit odd to impose such constraint. Then I realized that I didn't really need the HDR Histogram but rather a simplified version of our snapshots and desisted, but I could try to make it work
any thoughts on these limitations?
Gil Tene
@giltene
Sep 29 2015 14:47
If we do a JS version, it will need to handle the full range, because it will be the most likely version used for visualizing data from logs that would be produced by other versions. The main reason I want one is so that we can have nice things in browsers to view logs with. A secondary one is so that we can have an enhanced statsd that transports HdrHistigrams as values, and can summarize input as HdrHistiograms. In both cases, the values would arrive from outside of JS and can be in a wide range (e.g. DoubleHistogram in Java was motivated by a statsd use case).
As for the 32 bit thing: I'd go with an "always correct but usually fast" approach. Use the 32 bit operations when the values are small enough to fit, and revert to slower an uglier (but correct) code when a value too large to fit in a 32 bit integer is encountered.
The histogram's storage will also need to be carefully managed, since counts need to be able to hold values between 0 and Long.MAX_VALUE (2^63-1). Arrays of doubles are not good for that...
Michael Barker
@mikeb01
Sep 29 2015 20:00
If you can assume a modern browser, e.g. IE 10 or better then you could use typed arrays: https://developer.mozilla.org/en/docs/Web/JavaScript/Typed_arrays and split 64 bit values across 2 entries. You will probably end up with some slightly branchy code to deal with the >2^32 case.
Gil Tene
@giltene
Sep 29 2015 20:01
Would this also work on V8?
Michael Barker
@mikeb01
Sep 29 2015 20:41
Should do. Given V8 backs Chrome and has been supported since Chrome 7.
The ArrayBuffer that backs it is supposed to be a block of flat memory, with the intention of being more efficient that a standard JS array.