These are chat archives for HdrHistogram/HdrHistogram

28th
Apr 2016
Lee Campbell
@LeeCampbell
Apr 28 2016 03:13
@mjpt777 I have made my notes of the current format here - https://github.com/LeeCampbell/HdrHistogram.NET/wiki/V2-Log-Encoding
I can look into creating a EBNF def, but I have no experience in that.
w.r.t to the C# port (which is underway), I can add it as a feature.
@giltene I like the idea of the tags. I was about to do the three log files thing for wait, write and service times. Good to know I was on the right track.
Gil Tene
@giltene
Apr 28 2016 03:18
@LeeCampbell : Thanks for capturing the encoding details. Here are a couple of notes:
(1) The encoding we use is a ZigZag encoded LEB128-64b9B-variant. See the comment in ZigZagEncoding.java, which starts with "...the encoding used here diverges from the original" for details on how it diverges..." This difference is important to capture, as our encoding will use only 9 bytes in places where a ZigZag LEB128 may use 10 bytes.
Gil Tene
@giltene
Apr 28 2016 03:23
(2) I think the log file format and the encoded histogram format should be captured in two separate sections, such that the the encoding format is described separately and independently from the log file format (and the log file format refers to that description). Encoded histograms may appear in many places that are not log file lines...
Lee Campbell
@LeeCampbell
Apr 28 2016 03:37
Thanks @giltene . Have added issues to that repo. Will pull them over to https://github.com/HdrHistogram/HdrHistogram.NET as part of the port
Alec
@ahothan
Apr 28 2016 14:58
Given the (nice) proliferation of implementations, I think it would be nice to have a well known validation server that can take in encoded histograms and return some kind of diagnostic code to verify whether an encoding is correct or not.
Something as simple as a REST server that takes a JSON document containing things like implementation ID, encoded histogram ID and encoded histogram
All we need is a well known server to run that validation app and reachable in the internet - and a reference validation server (could be any language)
That way it would be possible to run automated sanity checks that gate the commits to the various repositories
Today we end up copying more or less the test code (along with the result templates) across all languages to verify that the implementation is correct.
Alec
@ahothan
Apr 28 2016 15:03
This adds to the need discussed above for some form of wrapper format that contains a bit more metadata than the simple encoded hdr histogram (which is natural because all apps using hdr histograms need to add their own metadata), might as well use a common one I think.
In our applications we use a simple json dictionary that has 1 field containing the histogram and a few others for our own metadata. One example is the wrapping of wrk2 results (a for of @giltene 's wrk2) by a python agent to transport HTTP latencies along with the other metrics (such as http requests per second,...)
Without such gating, it will be hard to verify language interop. In our case, the python implementation has to decode histograms from the C implementation across a cloud and everytime I update the repos I have to cross my fingers that something is not broken.