These are chat archives for HdrHistogram/HdrHistogram
nextbeforevis-a-vis ulp's availability in other languages, and I think that might actually be more correct. Might it not be possible that
x - ulp(x)is in fact smaller than the largest fp value smaller than
x? If so, that could lead to
ceil(x - ulp(x))being one smaller than
ceil(nextbefore(x)). I can implement it either way (https://docs.rs/ieee754/0.2.2/ieee754/trait.Ieee754.html has all I need), and as of Java 1.8 we have https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#nextDown-double- also.
x - Math.ulp(x) < Math.nextDown(x), e.g.
32.0 - ulp(32.0)=
x - ulp(x)and
nextDown(x), but it does seem like
nextDownis a better fit to compensate for the behavior in the comment in
fpCountAtPercentilebeing 1 ulp too big.
Math.doubleToRawLongBitsthat's the key in Java's case -- once you have that it's just some bit wrangling. You could write
Math.ulpyourself with a large dose of IEEE754 courage if you have the raw bits :)
64415.000000000015, whose previous FP value is
64415.00000000001which then rounds to
64416. That's in the next bucket, so the calculated value is
64447. Wouldn't this be a value that is too high?
(percentile / 100.0) * total countto
(percentile * total count) / 100.0, the test passes.
(percentile / 100.0) * total countcan be 2 ulps high. (In this instance,
x - ulpis the same as the immediately previous FP value, so that's not a problem here.)
total_countis more than a small number (100k in this case), it does kinda make sense to me that you wouldn't want to multiply the (imprecise) fp division of
percentile / 100.0by a large-ish number.
percentile * total_countwould also have error. In the worst case of
percentile = 100.0and
total_count = u64::max_value(), the ulp of that product is
262144. (That would then be divided by