These are chat archives for HdrHistogram/HdrHistogram
Sorry if that wasn't clear; I wasn't suggesting to
ceil() the input, only agreeing that it does make sense to
ceil() the product. (Is there a good name for the thing I've been describing as "quantile count", i.e.
quantile * count?)
I haven't dug very deep about
prev()-ing the quantile before multiplying by the count or
prev()-ing the product, though some quick tests seem to indicate that these tend to be identical:
let q_count_prev_before_mult = (quantile.prev() * length as f64).ceil(); let q_count_prev_after_mult = (quantile * length as f64).prev().ceil();
f64quantile so as to be fair) with the usual floating point arithmetic, and it finds lots of cases where the resulting error is 2 ulps. Maybe if I left it running for a while it would find 3 or 4 ulps, who knows.
quantile.prev() * count).ceil()seems to be equivalent to
(quantile * count).prev().ceil()inasmuch as 5 minutes of effort failed to find a case where they were different. I find them both difficult to rationalize, in that the best I can do is "it seems to work better if you throw a
previn there because FP multiplication where one argument is the FP equivalent of an integer is frequently just a touch too large".
+/- 1 ulpsince there are plenty of cases that are
+/- 2 ulp, and I am not confident that the upper bound of the error is 2.