These are chat archives for HdrHistogram/HdrHistogram

20th
Jul 2015
Alec
@ahothan
Jul 20 2015 20:06
Thanks for the pointer!
I know about the histogram add function but did not know about the histogram compressed format.
In my situation, I have a mixed language environment where the producer of histograms is wrk2 (C language) and the aggregator of histograms (consumer) is python.
I could have used a python to C wrapper but would prefer a fully native python library on the aggregator side (more portable).
Alec
@ahothan
Jul 20 2015 20:15
So a solution that uses a language specific wire encoding is not the best for me (for example the C code compresses an array of uint64_t counters which is not decodable from native python in a portable way, the base64 encoding on top of the compressed content is not a problem). It is more efficient generally but I think a solution that uses an open wire format is more inter-operable. I used JSON but could have used google protocol buffers to encode the significant information, I think most histograms will have very sparse counters, for example wrk2 uses 1 usec to 1 day range with 3 digits precision which yields 27*2048 counters. From what I have seen it looks like at most 10% of counters are non-zero so encoding only the non-zero counters should be quite efficient in size.
the other difference is the API I added does the addition of an encoded payload direct to the aggregation histogram (so does not create a new histogram for every new set of data) which somehow skips some instructions (plus the cost of compressing/decompressing)
Michael Barker
@mikeb01
Jul 20 2015 20:51
I'm not sure what you mean about the encoding not being portable. Is that a python restriction? The binary format will always be big endian regardless of platform.
Todd L. Montgomery
@tmontgomery
Jul 20 2015 20:52
for python, IIRC, struct can be used for packing and unpacking unsigned long long for uint64_t
goes all the way back to python 1.4 or so
Michael Barker
@mikeb01
Jul 20 2015 21:01
@ahothan The format is not intended to be language specific. The same format is used in both the Java and C implementations and the format is portable between them.
Gil Tene
@giltene
Jul 20 2015 21:12
@ahothan Sorry for joining the discussion a bit late. As @mikeb01 notes, we are striving for a common binary format that can be used in all implementations. This allows tools written in one language to process data logged by another. The current format used in the Java and C implementations (and by extension the erlang implementation, I think) ends up being very compact and easily transportable. The use of zlib compression ends up naturally benefiting from all the sparseness qualities of of typical histogram data, and is probably the most portable compression option available. The use of base64 in the log format is meant to allow data in histogram logs to be edited without requiring special binary editors.
To give you a feel for how compact the data ends up being, typical jHiccup log intervals take up only 150-250 bytes on avg. While occasional "busy data" intervals (intervals where large hiccups were experienced which resulted in a lot more buckets being populated) still tend to fit well below 1KB.
I've actually played around with trying to achieve better compression by leveraging sparseness (e.g. by performing an efficient single-ulong format run-length encoding ahead of the zlib step), and it turns out to not give you much of anything, since most LZW based compressions (including zlib) apply their own equivalent scheme.
Gil Tene
@giltene
Jul 20 2015 21:25
As for the sparseness argument for plain-text representations: while latency histograms are sometimes (and maybe most of the time) sparse, when they capture large latency events (such as hiccups, pauses, or deep queuing conditions) that sparseness tends to be significantly reduced (when no coordinated omission is in effect). E.g. when a "blip" of 2 seconds occurs in a system that normally serves 10K operations per second, 10K different values between 0 and 2,000,000usec will see recorded counts. A histogram that captures the latency distribution seen during such an event will NOT be sparse, but it will still be highly compressible (since most of those counts will fall in a very narrow integer range. A plain-text representation of counts at non-zero buckets (such as the JSON format you propose) would probably take up 80KB or so, while the compressed histogram will probably still fit in 1-2KB.
With all that said, having a human readable format in addition to compressed payload one may make sense to do. However, such a format should probably not expose the histogram's specific choice of bucket counts [that would make it not very human readable]. Instead, a format that lists pairs of values and counts (for non-zero values) is probably much more portable and human-readable at the same time.
Gil Tene
@giltene
Jul 20 2015 21:34
Separately, I like the idea of a pure-python (not just a C-wrapper) implementation of HdrHistogram. If you'd like to stand up a port that covers the commonly used APIs (e.g. those covered by the C port) and outputs to the common log format (in addition to whatever else) under HdrHistogram/hdrh_python or some such, I'd be happy to add it to the HdrHistigram org on github.
Alec
@ahothan
Jul 20 2015 23:12
thanks for all the great feedback, I have no doubt compression is the most efficient in all cases. I will try to get some bandwidth on implementing this wire format in python (although this would take me more time than I would have).
Michael Barker
@mikeb01
Jul 20 2015 23:21
@giltene I have a couple of changes that I'm going to push through on the Java implementation to make it explicit that the binary format is big endian and to support direct ByteBuffers.
Alec
@ahothan
Jul 20 2015 23:29
Pure python may require a bit more attention to overhead because it is inherently slower than C (and perhaps same as Java). Have anybody considered avoiding the cost of creating a new histogram instance just for the sake of adding it to another one? What I did was to simply pass the raw payload to my histogram function and let it directly add the buckets internally, this looked more optimal to me. In my use case, I have lots of histogram producers (hundreds) scattered in a cloud and they all send their histogram to an aggregator every X seconds (how the payload is being sent is irrelevant to this discussion but I can have cases where I need to aggregate hundreds of histograms per second, so much that I am thinking about tuning down the number of digits from 3 to 2 since I do not really need usec precision). So it is simpler in that i can control all my producers to have the same settings (min/max/digits) so I know that all internal buckets match exactly. I think that would be more efficient than converting all counters in range values since we'd have to do the reverse conversion at the aggregation side (in python this time). Using values instead of bucket indices as you suggest is indeed more universal (as all histograms do not have to look similar) but I'm worried about the extra cpu cycles that this would entail.