Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Michael Barker
    @mikeb01
    It would make implementing auto-resizing, which I'm yet to do, safer.
    Gil Tene
    @giltene
    "something strange" is going on with gitter for HdrHistogram/HdrHistogram. I have an e-mail form this morning with some new messages (from @ahothan and @ivantopo ), but they do not show up here.
    @ivantopo to your question about the sort of data I'm looking for: I'm looking for pretty much any server-side latency recording data that comes from "real-world" stuff. The purpose is to establish some baseline data sets to measure things like wire form and packed memory footprint with (since those footprints tend to be data dependent). Data collected across [many] intervals at clients (e.g. 1 second, 10 second, or maybe 1 minute) is probably more useful than data accumulated over longer periods across al intervals, as the client-side footprint would likely be for interval histogram collection, and only back ends will tend to deal with larger aggregated time scales.
    Ivan Topolnjak
    @ivantopo

    Hey @giltene, it took me quite some time to get back to this, but I got it: https://gist.github.com/ivantopo/33a2b04ec6c9fc005d194f74aa5c9f66

    That's a HdrHistogram log for 24 hours of data, in 1-minute intervals. The data was taken from one of our production endpoints that has a relatively predictable throughput, and even a couple big latency spikes. All the data was originally recorded using Kamon's HDR Histogram wrapper, configured with 2 significant value digits. I had to write a little script to query our backend, transform the data, and write it in a vanilla histogram I could dump on the HistogramLogWriter, but now that I have that it would be much easier if you guys need more data, different time intervals and so on

    Gil Tene
    @giltene
    Thanks @ivantopo ! Ill take a look at the data. Im very curious to see what the typical wire form sizes are, and what how PackedHistogram footprints look for this data. I habe high hopesfur packed histigram, and real world data will really help see if it delivers.
    filipe oliveira
    @filipecosta90

    @giltene from the discussion on the opentelemetry GH issue I can't understand whether it's required or not to have proto3 compatible histograms exporting.
    From the GH message:

    Because OTLP uses Google protobufs, OpenTelemetry will (I assume, speaking for myself) prefer to the interpret histogram data in a .proto representation, and "HDR Histogram" compatibility would mean being able to convert to and from a traditional HDR Histogram representation without loss and without significant requiring computation.

    Apart from it, should we join the effort of enabling benchmarks to the entire capabilities and the write benchmark described at https://github.com/dynatrace-oss/dynahist/blob/c4704414666377cb4aa18c20206934842212a473/charts/recording-speed.txt
    Do you believe that if we indeed have a double histogram variant across all languages the hdrhistogram can be picked? Are we aiming for the packed version?

    Gil Tene
    @giltene
    @filipecosta90 My current guess is that the OTEP will converge in an exponential rather than log-linear representation, and that we can hopefully have that representation include concepts (like avoiding a stated zero bucket size as a requirement, and allowing for auto-ranging, and possibly not specifying "base" but only "scale"). If those things go well, converting back and forth between e.g. a 2 decimal point accuracyHDRHistogram to a similar resolution OTLP (e.g. scale=7) will be straighforward, but involve some small loss of accuracy as the bucket count will be similar but the bucket boundaries will not be the same.
    Gil Tene
    @giltene
    This would be for DoubelHistogram, which covers the same capabilities that are worked on for the OTLP histogram. Got integeger histograms, converting to OTLP will be straightforward, but converting from it will only work well when no values below 1 are recorded...
    A discussion point for thsi group may be whether we want to consider a significant/radical "v3" change to HdrHistogram internals and APIs if OTLP solidifies on the current direction. E.g. we could have a new HdrHistogram internal representation that meets all the external requirements of the previous representations, but shifts to using exponential rather than log linear buckets. E.g. am exponential histogram with a "scale" of 7 is the semi-equivalent of a log-linear histogram with 2 decimal points. and an exponential histogram wqith a scale of 12 is the semi-equivalent of a log-lienar histogram with 3 decimal points. A new implementation (and wire format) would allow Hdrhistogram to be a fully OTLP-compatible thing, which will likely have value to many users...
    Gil Tene
    @giltene
    One big thing to consider in such an HdrHistogram 3.0 thing, beyind underlyign implementaion within the same APIOS, is whether we want to shift the APIs in some breaking way and make double precision floating point be the "normal/typical" value type, with integer-only (64 bit longs, 32 bit ints) being the special cases...
    Alec
    @ahothan

    One big thing to consider in such an HdrHistogram 3.0 thing, beyind underlyign implementaion within the same APIOS, is whether we want to shift the APIs in some breaking way and make double precision floating point be the "normal/typical" value type, with integer-only (64 bit longs, 32 bit ints) being the special cases...

    I agree, I think simplifying the HdrHistogram API is an important factor for wider adoption and the current footprint is rather large.

    Alec
    @ahothan
    We will still need to benchmark and compare the cost based on the type of counter used
    Gil Tene
    @giltene
    For the most commonly used precision (2 or 3 decimal points, which keans +/-0.5% or +/-0.05% relative error), I dont expect any difference in non-packed memory footprint, because the number of buckets covering a 2x range (e.g. between 2000 and 4000) end up being the same for log-linear and exponential. Exponential will have a somehwhat lower relative ertor for most values, but both still need the same number of buckets to guarantee the precision requirement as stated.
    Gil Tene
    @giltene
    the wire form or packed footprints may be slightly better statistically with exponential, but I’m doubtful itbwill be materisl (same number of buckets likely means, statistically, similar number of non-zero values, even if the boundaries of the buckets differ)
    Gil Tene
    @giltene
    perfirmance wise, on cpus with hardware double precision floating point support (which is virtually all server, desktop, or mobile pricessors these days), i expect the bit-bssed math that deduces the bucket index from the mantissa + exponent representation to be “similariy fast”. E.g. for integers, you convert the integer to a double (singie cycle op on most cpus) then use (exponent<<7) | (mantissa >> (52-7) to get the bucket index. For doubles, you can either go through an integer reorrsentation (which is what I currently do in DoubkeHistogram) or can do more direct conversion (but deal with auto ranging to cover bith negatuve or large positive exponents)
    Gil Tene
    @giltene
    The main benefits will be opentelemetry compatibility with no loss of precision on conversion to/from or merging with ipentelemetry histograms. And the DoubleHistogram implementation might get simpler.
    Gil Tene
    @giltene
    if we were willing to change the common Histogram type default to use double as the value type (covers the same or better recirding range than long does), we can end up with simplified internal representations.
    Alec
    @ahothan
    For most users I doubt exponential vs log linear will make any difference but it is worth considering this new implementation in HdrHistogram for open telemetry compatibility.
    As for wire format, it would be good to compare the current HdrHistogram format vs any protobuf based format (if there is even one available) in terms of codec speed and encoded size.
    Alec
    @ahothan
    open telemetry support for a transportable encoding such as the Hdrhistogram base64 blob will open up very interesting possibilities for native support in observability tools such as Grafana - for example being able to plot histograms and even iso latency curves from log or traces
    Gil Tene
    @giltene
    Yeah, it’s mostly the wire form / blob compatibility, with no conversion related precision loss, that I’m interested in. The underlying in-memory representation doesn’t matter: it’s a wash on both footprint and speed IMO.
    Gil Tene
    @giltene
    The proposed sparse histogram protobuf encoding will end up very similar to (equivalent to?) the combination of run-length enconding and zig zag that we already use. So the cookie and the compression are all that is left. We can have a v3.0 blob thats basically a compressed protobuf with a cookie in front.
    We fan probabky get the underlying tepresentation changed in a 3.0 format with no api changes at all, BTW, just a change to wire form. But I think it (a 3.0 version) would be a good opportunity to move from Integer to Double values by degault in the API, which is a breaking change in at least aome languages… e.g. Histogram in java would replace DoubleHistogram. And I’d probably drop ShortHistogram and IntHistogram now that we have PackedHistogram. And I may want to consider making packing/not-packing an option within Histogram, rather than a separate type…
    Gil Tene
    @giltene
    packing, auto-sizing, and auto-ranging are all comfiguration options (and might/shlould probably all be on by default).
    Gil Tene
    @giltene
    This is classic API “rationalization” after a long perrod of incremental addions of useful capabilities. When this all started, integer-value, fixed-storage-size (zero allocation for recording), user-specified range and balue fidelity were the sensicalstartimg points. We can still support all that, but over time options were added (in order of introduction): auto-sizing, Double values, auto-ranging, and packing. Bringing us to an as-clpse-to-zero-confuf-as-you-can-get practical implementation. If I were building the API from scratch, itbwould start with those all being the common and default case, and with predefined range, size, and zero-allocation all being optional configuration.
    Alec
    @ahothan
    I agree, I am one of the first victim of this rampage of add on features, I just could not keep up in the python code ;-)
    I think codec performance will be better for the current Hdrhistogram for most languages because it does not have to deal with the intermediate structures generated by the protobuf compiler.
    Histogram Codec is generally not as performance sensitive as an add record but there might be some high scale applications that juggle with lots of histogram blobs that need to be handled in near real time (eg merging large number of blobs very frequently in big data/ML).
    Gil Tene
    @giltene
    We can always have an additional more raw (v2.0 style, no protobuf parsing) blob if the performance delta warrants it.
    filipe oliveira
    @filipecosta90
    Hi there guys, good night,
    @mikeb01 , on a quick look over the percentile calculation function of the C lib I notice there's some quick extra 30% reduction we can win in a very easy manner:
    I've opened the PR showcasing the profile data and improvement changes:
    HdrHistogram/HdrHistogram_c#94
    Tomorrow EOD I can iterate over it again for further optimizations to see what pops up next :)
    Alec
    @ahothan

    Hi there guys, good night,
    @mikeb01 , on a quick look over the percentile calculation function of the C lib I notice there's some quick extra 30% reduction we can win in a very easy manner:
    I've opened the PR showcasing the profile data and improvement changes:
    HdrHistogram/HdrHistogram_c#94
    Tomorrow EOD I can iterate over it again for further optimizations to see what pops up next :)

    @filipecosta90 you're doing a terrific job optimizing this C code, looks good to me ;-)

    Michael Barker
    @mikeb01
    I'll try to have a detailed look over the code this week. A quick glance and it looks good. Nice investigation @filipecosta90
    filipe oliveira
    @filipecosta90
    thank you @ahothan and @mikeb01 . I've followed up on the fixes per the PR review.
    PS: I'm preparing a POC using hdr_histogram to keep track of Redis Internal latency and need to cut as much cpu-time as possible for reads/merge/ingestion so that it can be considered for Review.
    Alec
    @ahothan
    @filipecosta90 I'm curious about you using hdr histogram for this purpose, can you describe how this works better than using a traditional code profiler?
    Michael Barker
    @mikeb01
    I've merge the change as well as another bug fix that has been sitting in the queue for a while.
    @ahothan @filipecosta90 I would be interested in your thoughts on HdrHistogram/HdrHistogram_c#95. I want to complete some of the remaining features, but I would like to break the API in order to do it.
    1 reply
    Gil Tene
    @giltene
    @mikeb01 since you are looking at an api breaking change, and at adding double support, and I’m looking at what a 3.0 refactoring might look like with breaking changes and considering a direct-double-based (as opposed to integer based with double factor conversion) representation where integer values are “converted” to double rather than the other way around, we might want to sync up our thought before you go forward with a 2.0 based implementation…
    Michael Barker
    @mikeb01
    That sounds interesting. Let me spend a little bit of time spiking what I think what my implementation might look like and then we can share some ideas. I'm going to try and spend some time next week looking at it.
    Gil Tene
    @giltene
    If we use an exponential powers-of-2 scale, powers-of-2 precision representation (similar to open telemetry), as opposed to the current log-linear stuff, the math of deriving an index from a double FP value is interestingly simple bit extracts and shifts, much like the current math is for integers. I.e. no actual FP math is involved. Integers (longs) remain similarly simple (don’t even need integer to fp conversion, I think). So I think we can get the same speed as the 2.x stuff everywhere integers, and better speed (than 2.x) for doubles…
    It’s the API rethinking (fur Java at least) that I’m trying to get my head around. The simplest is to just make the values (the return types) doubles everywhere, and possibly accept either double or long as parameters. I’d probably get rid of DoubleHistogram, IntHistogram and ShortHistogram and consolidate everything under Histogram. (Still need concurrent and synchronized variants probably, but…)
    Michael Barker
    @mikeb01
    How does a powers-of-2 and powers-of-2 precision affect value the granularity of the values stored? I personally found that with the very basic power of 2 integer histogram not very useful.
    2 replies
    Raphael Salomao
    @raphaelsalomao_gitlab

    Hey folks! I'm having some issues while reading hlog files generated by the java hdrhistogram.
    My log files are generated with HistogramLogWriter.outputIntervalHistogram. I can read them fine with (java) HistogramLogReader, but I cannot seem to be able to load them with the Python equivalent.

    Here is what I see:

    reader = HistogramLogReader("histogram.hlog", None)
    hist = reader.get_next_interval_histogram()  # returns None
    print(hist.get_value_at_percentile(99.9))
    java_compressed_b64 = "HISTggAABHJ42u2az6tNURTH969z733P9dz3w..."  #  redacted histogram blob, can provide it if necessary
    histogram = HdrHistogram.decode(java_compressed_b64.encode())

    raises HdrCookieException

    histogram = HdrHistogram.decode(java_compressed_b64.encode(), False)

    raises zlib.error: Error -3 while decompressing data: incorrect header check

    I tried a few different things, maybe I'm dense but it's not immediately obvious to me what is going wrong, any advice on what I should look next?
    HdrHistogram/HdrHistogram_py#29 describes fairly closely what I'm experiencing (albeit with Rust impl).

    9 replies
    Marshall Pierce
    @marshallpierce
    That issue on the python impl is still open, so the bug in its decoding likely still exists.
    1 reply
    Raphael Salomao
    @raphaelsalomao_gitlab
    filipe oliveira
    @filipecosta90
    Hi there guys, good morning,
    I've opened a PR proposing the extension of latency stats from exposing simple averages per command on Redis to expose fixed percentiles and a CD of latencies ( only visual text formats are allowed ).
    Would appreciate your feedback @giltene @mikeb01 and all. Notice that the used HdrHistogram_c dep is a subset of the original version ( to avoid bloating the repo ).
    redis/redis#9462
    Notice that the latency interval we're interested in is between 1 microsecond and roughly 1 second ( we can consider everything above 1sec capped to it ). I'm proposing to maintain a value precision of 3 significant digits ( please state if you guys suggest otherwise ).
    Gil Tene
    @giltene
    I generally use 2 decimal points everywhere, and haven’t found many (any?) places that needed better. With 2 decimal points, the cost associated with covering well beyond what you think might be needed is much smaller, so e.g. using 1 nanosecond as the unit is only takes an extra 384-512 array entires (and gives you good resolution even for single digit microsecond numbers), and covering 1000 seconds rather than 1 second is similarly cheap.
    1 reply
    Jon Gjengset
    @jonhoo
    Hi folks! I posted over in the GitHub admin group (https://github.com/orgs/HdrHistogram/teams/admins/discussions/3), but it hasn't seen activity for a while so figured I'd try here too. It looks like HdrHistogram_rust hasn't been added to the Admin team, so I can't adjust its settings, which means I also cannot set up CI for it (which has apparently been broken since I moved the repo into the org) :sweat_smile: Would be great if it could be added (may have to be you @giltene, not sure) :+1:
    filipe oliveira
    @filipecosta90
    hi there @mikeb01 and @giltene , WRT to the C version oh hdrhistogram there's an open PR that mimics what was done on the go version. It's reducing by 7x the total time to compute percentiles.
    Do you guys think we can have it reviewed? HdrHistogram/HdrHistogram_c#107
    PS: similarly we can continue pushing the other optimizations we did on the go variation that will also benefit the C version
    Michael Barker
    @mikeb01
    Happy to receive PRs on those.
    filipe oliveira
    @filipecosta90
    thank you!
    Michael Barker
    @mikeb01
    @giltene What is the expected result of Histogram.getValueAtPercentile(0.0)? I would expect this to always be 0, but it is the lowestEquivalentValue of the first populated bucket in the current implementation.