Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Michael Barker
    @mikeb01
    One difference between the Java and C versions is that it in the C version is the callers responsibility to reset the histogram before passing it into be sampled.
    Gil Tene
    @giltene
    In the java version, the inactive is only temporary. It gets nulled out each time we return an interval histogram, and is fed either from an incoming recycled histogram or an allocation (if no incoming is supplied).
    The logic/reason for this is that it would be surprising (to java people at least) if the interval histogram they got and did not recycle started getting stomped and mutating after another interval get happens. Things like e.g. window stuff that wants to keep those histograms across a few interval gets won’t work as expected.
    Michael Barker
    @mikeb01
    With the C version now, it is only used when calling hdr_interval_recorder_sample, if the caller only uses hdr_interval_recorder_sample_and_recycle then it is never used.
    Gil Tene
    @giltene
    I make the recycling optional for people who are too lazy to do that and are ok just wastefully allocating on each get. There are many low tech and very-low-performance use cases of HdrHistogram where the “complexity” of the recycling would “get in the way”.
    (I’m being “polite” to the people who code like that)
    Michael Barker
    @mikeb01
    I'd almost be tempted to deprecate hdr_interval_recorder_sample. The nice thing about the hdr_interval_recorder_sample_and_recycle is that if you only track the return value in the calling code, you'll never run into an issue with it being changed unexpectedly.
    I'm a little bit meaner to those who are a bit lazier...
    Gil Tene
    @giltene
    Well, it’s C, you’re allowed and expected to be meaner.
    I agree that for C, and auto-implicitly-allocating get would not be a good idea. No GC to make that “just work”.
    Gil Tene
    @giltene
    So deprecating the old api would probably be the right thing.
    Gil Tene
    @giltene
    I’ve updated the various Java recorders to take an optional enforceContainingInstance Boolean parameter on getIntervalHistogram().
    Nitsan Wakart
    @nitsanw
    Hi! is there any justification for the almost full duplication between the AbstractHistogramLogReader and the HistogramLogReader?
    I'm looking to tackle a couple of issue in this area, in particular making the log read cloasable and making it easier to handle errors.
    Michael Barker
    @mikeb01
    Relased 0.9.11 version of the c hdr histogram with support for both Win32 and Win64.
    Todd L. Montgomery
    @tmontgomery
    :thumbsup:
    Jon Gjengset
    @jonhoo
    @giltene we should probably migrate over the HdrHistograms to use travis-ci.com now that .org is being deprecated: https://docs.travis-ci.com/user/migrate/open-source-repository-migration/
    alternatively we could choose some other CI like Azure Pipelines, Circle CI, etc.
    related to that, I also just sent a request to enable codecov integration for the rust repository
    Alec
    @ahothan
    Hi @mikeb01, was wondering if anybody has been compiling hdr histogram C using a c++ compiler lately?
    I got several warnings related to void * implicit casting which is allowed with C compiler and results in warnings with c++ compiler
    here is the list:

    ../../src/hdrh/hdr_histogram.c: In function ‘int hdr_init(int64_t, int64_t, int, hdr_histogram*)’:
    ../../src/hdrh/hdr_histogram.c:362:61: error: invalid conversion from ‘void
    ’ to ‘int64_t {aka long int}’ [-fpermissive]
    counts = calloc((size_t) cfg.counts_len, sizeof(int64_t));
    ^
    ../../src/hdrh/hdr_histogram.c:368:55: error: invalid conversion from ‘void’ to ‘hdr_histogram’ [-fpermissive]
    histogram = calloc(1, sizeof(struct hdr_histogram));
    ^

    ../../src/hdrh/hdr_histogram_log.c:862:15: error: invalid suffix on literal; C++11 requires a space between literal and identifier [-Werror=literal-suffix]
    file, "%.3f,%.3f,%"PRIu64".0,%s\n",
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_decode_compressed_v0(_compression_flyweight, size_t, hdr_histogram**)’:
    ../../src/hdrh/hdr_histogram_log.c:467:60: error: invalid conversion from ‘void
    ’ to ‘uint8_t {aka unsigned char}’ [-fpermissive]
    if ((counts_array = calloc(1, (size_t) counts_array_len)) == NULL)
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_decode_compressed_v1(_compression_flyweight, size_t, hdr_histogram**)’:
    ../../src/hdrh/hdr_histogram_log.c:567:60: error: invalid conversion from ‘void
    ’ to ‘uint8_t {aka unsigned char}’ [-fpermissive]
    if ((counts_array = calloc(1, (size_t) counts_array_len)) == NULL)
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_decode_compressed_v2(_compression_flyweight, size_t, hdr_histogram**)’:
    ../../src/hdrh/hdr_histogram_log.c:664:60: error: invalid conversion from ‘void
    ’ to ‘uint8_t {aka unsigned char}’ [-fpermissive]
    if ((counts_array = calloc(1, (size_t) counts_limit + 9)) == NULL)
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_log_write(hdr_log_writer, FILE, const hdr_timespec, const hdr_timespec, hdr_histogram)’:
    ../../src/hdrh/hdr_histogram_log.c:852:61: error: invalid conversion from ‘void
    ’ to ‘char’ [-fpermissive]
    encoded_histogram = calloc(encoded_len + 1, sizeof(char));
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_log_encode(hdr_histogram
    , char)’:
    ../../src/hdrh/hdr_histogram_log.c:1169:65: error: invalid conversion from ‘void’ to ‘char’ [-fpermissive]
    encoded_histogram_tmp = calloc(encoded_len + 1, sizeof(char));
    ^
    ../../src/hdrh/hdr_histogram_log.c: In function ‘int hdr_log_decode(hdr_histogram
    , char, size_t)’:
    ../../src/hdrh/hdr_histogram_log.c:1193:65: error: invalid conversion from ‘void
    ’ to ‘uint8_t {aka unsigned char}’ [-fpermissive]
    compressed_histogram = malloc(sizeof(uint8_t)*compressed_len);

    Michael Barker
    @mikeb01
    No I haven't tested that. If you could throw that into a github issue, I'll get those cleared up this weekend. Thanks.
    Alec
    @ahothan
    @mikeb01 I have submitted a PR for this: HdrHistogram/HdrHistogram_c#66
    I have one more question related to hdr_time.h,
    Alec
    @ahothan
    In my use case, I will be compiling and linking hdrh-c into a c++ app that has some restrictions in dependency runtime libraries (this app is an open source traffic generator). It only uses hdrh-c to create/fill up histograms and encode then. For this use case, is there a need for hdr_time.h
    Alec
    @ahothan
    the issue i have is hdr_time.h is being pulled by hdr_histogram_log.h and therefore the app fails to load because one specific target does not have the clock_gettime library
    so I'm trying to see if it is possible to separate out the clock_gettime() call from any object file that we use on our "fast path"
    Alec
    @ahothan
    @mikeb01 it looks like , hdr_getnow() and hdr_gettime() (which in turn call the problematic clock_gettime()) are only used by test functions. While hdr_timespec_as_double() and hdr_timespec_from_double() which are needed by hdr_histogram_log.c do not have any issue per se. So it looks like it would make sense to split these in 2 files - although that would potentially cause a backward incompatibility in any user code that includes hdr_time.h. For now I'm just going to remove hdr_gettime() and hdr_getnow() from my copy of hdr_time.c
    Michael Barker
    @mikeb01
    I'll pull your PR and have a dig into these a bit later this week.
    I'll see if there is a way to fall back, e.g does the platform without clock_gettime have gettimeofday?
    BTW some platforms need to link to librt (-ltr) to get clock_gettime.
    Alec
    @ahothan
    @mikeb01 thanks for the merge. We will not be able to add librt as we want to keep the size of the traffic gen as small as possible.
    For now I just commented out the 2 funcgtions in hdr_time.c and that works fine
    Baruch Even
    @baruch
    I'm interested in using HdrHistogram in Dlang, I've implemented the basics and want to test it, I couldn't find a test sequence to work with. Is there anything like that? (like a test vector for encryption)
    r_mohan
    @r_mohan_twitter
    ```
    public void testAddWithAutoResize() {
    DoubleHistogram histo1 = new DoubleHistogram(3);
    histo1.setAutoResize(true);
    histo1.recordValue(6.0);
    histo1.recordValue(1.0);
    histo1.recordValue(5.0);
    histo1.recordValue(8.0);
    histo1.recordValue(3.0);
    histo1.recordValue(7.0);
    DoubleHistogram histo2 = new DoubleHistogram(3);
    histo2.setAutoResize(true);
    histo2.recordValue(9.0);
    DoubleHistogram histo3 = new DoubleHistogram(3);
    histo3.setAutoResize(true);
    histo3.recordValue(4.0);
    histo3.recordValue(2.0);
    histo3.recordValue(10.0);
        DoubleHistogram merged = new DoubleHistogram(3);
        merged.setAutoResize(true);
        merged.add(histo1);
        merged.add(histo2);
        merged.add(histo3);
    
        assertEquals(merged.getTotalCount(),
                histo1.getTotalCount() + histo2.getTotalCount() + histo3.getTotalCount());
        assertEquals(1.0, merged.getMinValue(), 0.01);
        assertEquals(10.0, merged.getMaxValue(), 0.01);
    }
    The delta in the assertEquals is what I don't get.
    How do these simple values lose that precision ?
    What is I need to store values like 0.81, 0.21 and so on ? How do I understand that delta when I get the true min. and true max. ?
    Gil Tene
    @giltene
    In your code above, the required precision of the DoubleHistograms created is set to 3 decimal points, which means that the histograms are required to maintain this precision (or better), and are allowed to track values within those constraints. A required precision of 3 decimal points allows values that are within 0.1% of one another to be considered equivalent. This includes any recorded values, and this allowed assumption is used in the internal data structures that capture information about the recorded values To maintain consistency, the Maximum is reported as the largest value equivalent to the largest value recorded, and the Minimum is reported as the smallest value that is equivalent to the smallest value recorded.
    Given a histogram you can determine e.g. lowestEquivalentValue(), highestEquivalentValue(), nextNonEquivalentValue(), valuesAreEquivalent(), for any values you want. As long as the Max value reported is equivalent to the actual largest value recorded, things are correct.
    r_mohan
    @r_mohan_twitter
    Thanks. AtomicHistogram and WriterReaderPhaser. Doesn't the internal datastructure ensure concurrency already ? I miss that point.
    Gil Tene
    @giltene
    You should not (typically) need to use WriterReaderPhaser. Look at the Recorder and SingleWriterRecorder, which make use of it under the hood to provide you with the sort of view into stable histogram data in a concurrent value recording environment.
    AtomicHistogram provides recording atomicity but does not support auto-sizing (must be statically sized). ConcurrentHistogram supports concurrent value recoding even under auto-sizing (and support shifting needed for ConcurrentDoubleHistogram, which uses ConcurrentHistogram under the hood).
    Kevin Binz
    @KevinBinz
    Is there an easy way to get/produce Windows & Linux Python 3 .whl files for https://github.com/HdrHistogram/HdrHistogram_py? They are currently missing from https://pypi.org/project/hdrhistogram/0.6.1/#files Sorry if this isn't the right place for this question.
    Gil Tene
    @giltene
    Hey folks! My new (in the works as weekend work for a while now) packed array thing is now up in the java version. It is behind the new PackedHistogram, PackedConcuurentHistogram, double variants if the same, and a ”packed” option on all recorder types. Would love to get this poked and prodded at critically here.
    Alec
    @ahothan
    @KevinBinz I can try to publish whl files for Linux but I dont have any Windows server. If you have bandwidth to add it to the repo ci/cd, I'd be happy to merge your contribution.
    Alexandre Victoor
    @Alex_Victoor_twitter
    Hello @giltene !
    I have spend some time reading at the java code. I guess I will start soon the packed JS implementation. Since JS is single threaded, I do not need to handle concurrent access and AbstractPackedArrayContext might become really simple... I guess I do not even need the header of the nonLeafEntry structure.
    At this point I have a couple of questions:
    • What is the math to get the packed threshold MAX_SUPPORTED_PACKED_COUNTS_ARRAY_LENGTH ?
    • Same question for the max number of blocking ops (74) when adding a value in ConcurrentPackedLongArray ?
    Marshall Pierce
    @marshallpierce
    neat, will have a go at the rust version of PackedHIstogram when I have a moment or three