Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Olavo Masayuki Machado Shibata
    @olavoshibata
    I want to use Hollow for a small set of data. It is around 50 entries. The idea of using Hollow is because I don't want to restart my application all the time I change this data. This data changes weekly. Would Hollow be a good solution for that?
    milk
    @milk89676173_twitter

    Is the latest stable version for Hollow really 3.0.1, as the README states, or is 5.0.8 considered stable?

    Additionally, has anyone upgraded from any of the 3.x versions to 5.x? Did you have to migrate to a new namespace or produce a new snapshot, or is it safe to upgrade in-place with an existing producer-consumer already running?

    zaverrabadiya
    @zaverrabadiya

    Hello,

    Question about push notification for delta updates, referencing from the Hollow.how "When your AnnouncementWatcher is initialized, you should immediately set up your selected announcement mechanism -- either subscribe to your push notifications or set up a thread to poll for updates."

    How do I subscribe to pus notification? the HollowReference project polls (setupPollingThread()) the updates I believe as per the below code snippet:

    public DynamoDBAnnouncementWatcher(AWSCredentials credentials, String tableName, String blobNamespace) {
            this.dynamoDB = new DynamoDB(new AmazonDynamoDBClient(credentials));
            this.tableName = tableName;
            this.blobNamespace = blobNamespace;
            this.subscribedConsumers = Collections.synchronizedList(new ArrayList<HollowConsumer>());
    
            this.latestVersion = readLatestVersion();
    
            setupPollingThread();
        }

    I could not find an example of push notification, any guidance/pointer will be helpful!

    Appreciate help in advance!

    zaverrabadiya
    @zaverrabadiya
    Ping! can someone answer above mentioned question? Thanks!
    Mike Muske
    @mikemuske
    @rpalcolea Can you guys take a look at this PR? Netflix/hollow#512
    Mike Muske
    @mikemuske

    @jkade v4.9.2 is the latest. This project is still actively developed and used widely at Netflix. I agree the docs could use some more attention.

    This was the last message I could find from a Netflix person, and it is 8+ months old. Is anyone from Netflix still monitoring this channel?

    @dkoszewnik :point_up:
    Drew Koszewnik
    @dkoszewnik
    @mikemuske taking a look now.
    I'm assuming you are blocked on this? Should we push a release out for you immediately?
    Drew Koszewnik
    @dkoszewnik
    @mikemuske I have merged the PR and will cut a release imminently.
    Erich Weidner
    @erichw
    Thanks @dkoszewnik
    Drew Koszewnik
    @dkoszewnik
    Absolutely, thanks for the fix. v5.2.2 releasing now.
    Erich Weidner
    @erichw
    That one was a tough one to find and only occurred on datasets where our producer was running with 100+GB which made debugging very difficult.
    Drew Koszewnik
    @dkoszewnik
    @erichw yes I would imagine that took a lot of detective work. Nice work and thanks again for tracking it down :).
    Mike Muske
    @mikemuske
    @dkoszewnik thank you!
    Dillon Beliveau
    @Dillonb
    Hey everyone - I didn't quite feel this was issue worthy, but I've been trying out Hollow on the AWS Graviton aarch64 instances, and noticed an unaligned bus access in FixedLengthElementArray, possible on x86_64, hardware exception on aarch64. Judging by the comments in that class it seems intentional. I'm curious if there are any plans on addressing this on the roadmap, or if it has been looked into before?
    Dillon Beliveau
    @Dillonb
    In the meantime I'll see what I can come up with
    Sunjeet
    @Sunjeet
    @Dillonb we did some testing on ARM last year and we observed SIGBUS crash in unaligned access in FixedLengthElementArray. The failure seemed to only occur in the delta application path, so we were able to run microbenchmarking on ARM vs x86
    I can share our results from the microbenchmarks we ran if you're curious
    We didn't get to identifying the root cause of SIGBUS on arm, and we don't have anything on the roadmap for arm support
    Me and some other folks here would be keen to hear about what you can find though :)
    Dillon Beliveau
    @Dillonb

    @Sunjeet it seems to be one/all of these three calls to putOrderedLong() in FixedLengthElementArray that's causing the crash I'm seeing at least: https://github.com/Netflix/hollow/blob/master/hollow/src/main/java/com/netflix/hollow/core/memory/encoding/FixedLengthElementArray.java#L194-L200

    ARM doesn't support reading/writing unaligned values - the solution I believe would be replacing those calls with plain old accesses to the array. It would negatively impact performance, though, so ideally we could somehow use the existing path on platforms that support it, and the new path only on platforms that do not

    Mike Muske
    @mikemuske
    @dkoszewnik I saw your note about an additional spot where there may be an overflow. I've looked at it a little and it appears to me like you're right. https://github.com/Netflix/hollow/blob/7ff25c9b3113d731341ca203ee81a46d7ab46cdc/hollow/src/main/java/com/netflix/hollow/core/write/HollowListTypeWriteState.java#L226-L227
    Did you confirm it's not a problem?
    Dillon Beliveau
    @Dillonb

    I'd love to see anything you're willing to share about your testing on ARM in addition to this PR Netflix/hollow#503 though :)

    I'll try to spend some time this weekend hacking on this to see if I can get something together.

    Drew Koszewnik
    @dkoszewnik
    @mikemuske after posting that, I realized the integer written there is the number of 64-bit longs required to write all of the element data, not the number of elements. So you should be ok there.
    Mike Muske
    @mikemuske
    well, that will be a little smaller, but won't it still exceed 32 bits?
    For example, the dataset I'm attempting to load now has 6.6 billion total array elements, 6.6 billion x 33 bits = 217,800,000,000, and dividing by 64 gives 3,403,125,000 which still overflows Integer.MAX_VALUE
    Drew Koszewnik
    @dkoszewnik
    Your references are 33 bits each?
    Mike Muske
    @mikemuske
    i was thinking they'd have to be... but maybe that's where i went wrong
    Drew Koszewnik
    @dkoszewnik
    The number of bits required for each element will only be the number required to represent the max ordinal of the referenced type in your dataset.
    Mike Muske
    @mikemuske
    got it, so is that 29 bits then?
    Drew Koszewnik
    @dkoszewnik
    So it depends what the cardinality of your referenced type is, if you only have 1,000 unique elements, then you'll need 10 bits.
    Mike Muske
    @mikemuske
    i see
    Drew Koszewnik
    @dkoszewnik
    6.6 billion seems very large, pushing the limits here -- maybe there's some way to remodel the data?
    Mike Muske
    @mikemuske
    I guess in this case we are expecting about 121 million strings, so only need 27 bits, but it will still overflow.
    Yeah, it's a ridiculous dataset. I think we can shard the arrays into 4 separate spaces, so that will make it fit. But, it seems like we could fix this anyway.
    Sunjeet
    @Sunjeet
    Logged this issue with the little data that we were able to gather from our ARM testing Netflix/hollow#517 @Dillonb
    Dillon Beliveau
    @Dillonb
    Thanks!
    Dillon Beliveau
    @Dillonb
    Hey @Sunjeet, curious to hear your thoughts on Netflix/hollow#518 when you have a chance
    mailsanchu
    @mailsanchu
    Hi All, Is there a way to get the written object using mapper.add(node) back?
    1 reply
    MichaƂ Olejnik
    @blekit
    Hello guys, probably a silly question from me. I'm trying to build Hollow and I'm getting a few errors related to missing sun.misc.Unsafe (neither building from console nor from IntelliJ works). I currently have JDK 11 and JDK 15 installed on my machine. Can someone point me towards change I need to make to build the project?