These are chat archives for HdrHistogram/HdrHistogram

7th
Oct 2015
Gil Tene
@giltene
Oct 07 2015 00:36
There is this question about how people use HdrHistogram on mechanical sympathy: https://groups.google.com/forum/#!topic/mechanical-sympathy/gfTGw580IrU
Thought I'd wait for other people to post first.
@ahothan : the questions bout using it for benchmarking (or monitoring?) with multiple nodes is probably one that you can relay some direct personal experience for, right?
Alec
@ahothan
Oct 07 2015 05:00
@giltene I just posted a description of what I did. With the advent of cloud computing and IoT, there is no doubt that the need for measuring distributed latencies is going to rise. A big change of era.
Alec
@ahothan
Oct 07 2015 05:05
I knew pretty well ASN.1 (I had to work on a portable version of an ASN.1 compiler in C). It is true that many technologies are transient, but I think JSON is going to last a lot longer than ASN.1 because it is much simpler to use and more widely adopted (ASN.1 was pretty much only used by the initiated and by gov. projects). As for text, yeah it's not going away any time soon but scraping text is going to rebuke many people - unless you provide a library to scrape your text in every language.
It was easy for me to add domain specific fields in the json envelope along with the encoded histogram. It is a lot less attractive to have to add my own metacomments in the log text, and write the scraper at the other end.
Alec
@ahothan
Oct 07 2015 05:16
As for predefining fields. they are very much domain specific so not sure if it is worth standardizing there (other than maybe the few obvious ones you pointed out).
Gil Tene
@giltene
Oct 07 2015 06:39
@ahothan : So what do you think about a convention for carrying JSON encoded metadata in (single line) comment lines in the log format?
Alec
@ahothan
Oct 07 2015 14:14

@giltene less custom scraping is always a good step, but I have to ask, are you concerned about preserving backward compatibility?
The V1.2 log has a mix of metacomments and one liners with timestamps+encode

[Logged with jHiccup version 2.0.7-SNAPSHOT]

[Histogram log format version 1.2]

[StartTime: 1441812279.474 (seconds since epoch), Wed Sep 09 08:24:39 PDT 2015]

"StartTimestamp","Interval_Length","Interval_Max","Interval_Compressed_Histogram"
0.127,1.007,2.769,HISTFAAAAEV42pNpmSzMwMCgyAABTBDKT4GBgdnNYMcCBvsPEBEJISEuATEZMQ4uASkhIR4nrxg9v2lMaxhvMekILGZkKmcCAEf2CsI=
...
Sorry to be blunt but this looks like a ragtag syntax for any newcomer, I am sure there is some history behind this.
Some DS metadata will have to be tied to every encode (same level as the start timestamp...). This will force to make the metacomments position dependent (like before every encode line). Something like:

[Logged with jHiccup version 2.0.7-SNAPSHOT]

[Histogram log format version 1.2]

[StartTime: 1441812279.474 (seconds since epoch), Wed Sep 09 08:24:39 PDT 2015]

[Json { ... domain specific fields ... }]

"StartTimestamp","Interval_Length","Interval_Max","Interval_Compressed_Histogram"

[Json { ... per encode DS fields }]

0.127,1.007,2.769,HISTFAAAAEV42pNpmSzMwMCgyAABTBDKT4GBgdnNYMcCBvsPEBEJISEuATEZMQ4uASkhIR4nrxg9v2lMaxhvMekILGZkKmcCAEf2CsI=

[Json { ... per encode DS fields }]

1.134,0.999,0.442,HISTFAAAAEJ42pNpmSzMwMAgxwABTBDKT4GBgdnNYMcCBvsPEBEWLj45FTExAT4pBSEBKa6UkAgBi1uM7xjfMMlwMDABAC0CCjM=

as compared to:
{
"author": "jHiccup version 2.0.7-SNAPSHOT",
"version": "1.2",
"start_time_epoch": 1441812279.474,
"start_date": "Wed Sep 09 08:24:39 PDT 2015",
... DS fields ...
"histograms": [
{
"start": 0.127,
... DS fields ...
"encode": "HISTFAAAAEV42pNpmSzMwMCgyAABTBDKT4GBgdnNYMcCBvsPEBEJISEuATEZMQ4uASkhIR4nrxg9v2lMaxhvMekILGZkKmcCAEf2CsI="
},
etc...
]
}
or even
{
"author": "jHiccup version 2.0.7-SNAPSHOT",
"version": "1.2",
"start_time_epoch": 1441812279.474,
"start_date": "Wed Sep 09 08:24:39 PDT 2015",
... DS fields ...
}\n
{
"start": 0.127,
... DS fields ...
"encode": "HISTFAAAAEV42pNpmSzMwMCgyAABTBDKT4GBgdnNYMcCBvsPEBEJISEuATEZMQ4uASkhIR4nrxg9v2lMaxhvMekILGZkKmcCAEf2CsI="
}\n
etc...

There is a large number of open source tools around json: json schemas, json viewers, json editors, and now a lot of tools around REST modeling with json content. These make it hard today to go agaist the flow.
Another intangible factor is the coolness factor, not for people like you and me but you have to say that with millenials that is important for adoption and clearly scraping text does not exactly play into that (every potential user of HdrHstogram encodes will have to add DS fields to it)...
Lastly it must be tempting to have to deprecate all the text scraping code in all current apps and replace with 1 json loads(), and the text encoding code with 1 json dumps(), the rest of the changes is practically mechanical.
Perhaps, the number of tools that use the current V1.2 format will be very small compared to the potential number of future tools that will use it to make it worth upgrading the syntax?

just hate this gitter editor, it screwed up again he formatting with '#['