by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    yulibanpenagos
    @yulibanpenagos

    Hello all. I'm trying to use hollow to put a large number of objects in memory starting from a file containing a lot of json objects. To avoid loading the json in memory (which is big) I'm using Reactx to stream the reading of the file and the parse from json to my dto model. Like this:

    producer.runIncrementalCycle(writeState -> {
        p.generateStyleStreaming(jsonPath).subscribe(
                e -> {
                    writeState.addIfAbsent(e);
                }
        ).dispose();
    });

    I'm using the hollow incremental producer BTW.
    The p.generateStyleStreaming(jsonPath) is the flow reading the file and parsing the json.
    However, even with a large amount of heap (5048) it's throwing Java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space

    The input file is large, about 24GB and each json object is big as well, so I was wondering if there is some constraint related to it.
    There are about 10 million json objects in the input file BTW.
    yulibanpenagos
    @yulibanpenagos
    If I try to process only the json it works w/o problem
    yulibanpenagos
    @yulibanpenagos
    *Disclaimer: Reactx -> RxJava
    balatbn
    @balatbn
    Does hollow support primitive arrays? Like int[], long[]?
    I can use List<Integer> and List<Long>, but my understanding is that this will create new object for each of the value in the list. Is that assumption correct?
    Drew Koszewnik
    @dkoszewnik
    @balatbn primitive arrays are not supported, and your understanding is correct.
    balatbn
    @balatbn
    ^Thanks
    Bryan Jang Kim
    @jjangsam
    Will findKeysWithPrefix result in O(1) lookup? Is hashKey lookup always preferred choice than findKeysWithPrefix in terms of performance?
    sujithramanathan
    @sujithramanathan
    Hi,
    Is there a way to create snapshots periodically ?
    Bryan Jang Kim
    @jjangsam
    @dkoszewnik It's my understanding that a change to a child object ripples up as a change to a parent object. But the following lines of code shows that the parent keeps its own set of child deltas.
                    producer.runIncrementalCycle(writeState -> writeState.addOrModify(new ImmutableMovie(1, "movie1", Set.of(new ImmutableActor(1, "actor1"), new ImmutableActor(2, "actor2")))));
            producer.runIncrementalCycle(writeState -> writeState.addOrModify(new ImmutableMovie(2, "movie2", Set.of(new ImmutableActor(1, "actor1"), new ImmutableActor(2, "actor2 modified")))));
    
            consumer.triggerRefresh();
            UniqueKeyIndex<com.disney.dtci.vidsearch.hollow.producer.generated.ImmutableMovie, Integer> uki = com.disney.dtci.vidsearch.hollow.producer.generated.ImmutableMovie.uniqueIndex(consumer);
            System.out.println(hollowRecordJsonStringifier.stringify(uki.findMatch(1)));
            System.out.println(hollowRecordJsonStringifier.stringify(uki.findMatch(2)));
    The above results in
    {
      "id": 1,
      "title": "movie1",
      "actors": [
        {
          "actorId": 2,
          "actorName": "actor2"
        },    {
          "actorId": 1,
          "actorName": "actor1"
        }
      ]
    }
    {
      "id": 2,
      "title": "movie2",
      "actors": [
        {
          "actorId": 1,
          "actorName": "actor1"
        },    {
          "actorId": 2,
          "actorName": "actor2 modified"
        }
      ]
    }
    Shouldn't both movie have actorId 2 with actorName "actors2 modified"
    Elisa Natalie
    @octopusgarden
    Why is the latest stable version in documentation (https://github.com/Netflix/hollow) is 3.0.1 while in maven repository (https://mvnrepository.com/artifact/com.netflix.hollow/hollow) we have version 4.7.x? Is it just because the documentation is not updated?
    Bryan Jang Kim
    @jjangsam
    I think the doc is outdated
    adwu73
    @adwu73
    Hi,we are using Hollow to build a trello like application,we store all the cards in hollow, and use hollow hash index to filter cards out,now we are experiencing some load performance issue now. We w
    When we have 1
    user, we can assemble a card from Hollow within 100ms,however, when we have 50 concurrent users,the time will grow to 1s for each user.
    Is there any lock mechanism when we are reading data out of hollow? Will use type api or caching helps? Thank very much?
    krishnaram
    @krishnaram
    Hi I am getting this exception
    java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException
    at com.netflix.hollow.core.write.HollowWriteStateEngine.resetToLastPrepareForNextCycle(HollowWriteStateEngine.java:265)
    Please help me in this
    damianChapman
    @damianChapman
    Is it possible with the latest version of Netflix Hollow that you can create an index to return all results for an array field that has zero size?
    Deepak Garg
    @deepak0917
    we recently updated to 3.8.1 from 3.7.7 as we wanted to fix issue mentioned here -> Netflix/hollow#431
    but we have started noticed this error " Attempting to apply a delta to a state from which it was not originated!" after the upgrade.
    our service has been running since 3 months but we noticed this error only after the upgrade. we rolled back to 3.7.8 but still the issue is happening. please help @dkoszewnik @toolbear
    Drew Koszewnik
    @dkoszewnik
    Hi @deepak0917
    What this means is that either the delta your clients are attempting to apply did not originate from the state which they currently have in memory (e.g. it is tagged incorrectly), or the state is somehow corrupted in the memory on the clients.
    The most straightforward way to get out of this situation is to restart the consumers so that they can load a more recent snapshot, then continue to apply deltas from there. Is that possible in this case?
    If that is not possible, then you need to determine what state the clients are actually in, then restart your producer and restore from that state. The next published state will create a delta from the consumers' current state and they will consume it and resume progress.
    Bryan Jang Kim
    @jjangsam
    Hi Is there a performance downgrade for having String as primary key? I have about 200k entries and seeing slower lookup time using XXX.uniqueIndex(consumer) method. Each look up taking more than 100ms. The primary key is pretty lengthly string, roughly about 50 characters. Please help @dkoszewnik @toolbear!
    Bryan Jang Kim
    @jjangsam
    Actually I found my answer from the doc "Retrieval from an index is extremely cheap, and indexing is (relatively) expensive. You should create your indexes when the HollowConsumer is initialized and share them thereafter. Indexes will automatically stay up-to-date with the HollowConsumer"
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    Hello, could someone confirm if the currentVersion metric of the hollow producer is supposed to be changed even when there is no change in state? It seems wrong to me but I'm not sure if I fully understand the system. I have opened this PR to address this https://github.com/Netflix/hollow/pull/458/files
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    Clarification for the above: we want to set up an alert for consumers that are out of sync with the producer. For that we're comparing the consumer's version metric with the producer's. However the producer's version is not reliable as it changes on each cycle even when there's nothing to publish
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    Any thoughts on the above ^? The change is minor. Should I compile a report of our current configuration and the behavior we observe in order to more easily validate whether this is an issue?
    Drew Koszewnik
    @dkoszewnik
    @AlexandruGhergut the cycle version does need to update in the producer even when no change in the underlying data occurs. This is so that we can understand that progress is being made, even if no changes are occurring. For example, we store log lines tagged with each cycle version in a system like elasticsearch, then we can peruse what is happening in the system in a dashboard that uses the data in this system. Rather than change the meaning of this field, how about adding another field that indicates the last announced version, then using that to drive your alert?
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    @dkoszewnik thank you for your response. I was thinking of the currentVersion as the last published version but it makes perfect sense to associate a new version to each cycle for tracking purposes. Another field would do the job. I used your suggestion and opened a new PR Netflix/hollow#459
    Viktor Nyström
    @viqtor
    how does one work with compound keys and UniqueKeyIndex? the generated primary key index supports compound primary keys as explained in https://hollow.how/indexing-querying/#compound-primary-keys but i cant see how the UniqueKeyIndex would and its referenced in the deprecation notice on the previous class
    does that mean that compound keys are also deprecated?
    Rich Bolen
    @richbolen
    Does hollow support less than or greater than queries? We have a need to query our data set with a calendar date. If it is >< two date fields in the data, return a record.
    Viktor Nyström
    @viqtor
    as a follow up on my previous post. if i use the deprecated generated primary key index and don't specify the field paths for the index the order of the fields are reversed?
    Miklos Szots
    @smiklos

    Hi all,

    Is it a valid usecase for Hollow to have a job peridocally take data from a small table and publish that? The focus would be on having the Hollow Producer be a job in kubernetes as I see no reason for it to be a 24/7 app running

    Viktor Nyström
    @viqtor
    @smiklos by all means. sounds reasonable to me at least. just make sure to restore your producer on startup in the job first https://hollow.how/producer-consumer-apis/#restoring-at-startup
    Miklos Szots
    @smiklos
    @viqtor Thanks. I was aware of this but the docs said that the ideal usage is to reuse the producer. I guess from a metric collection perspective, that's easier.
    Miklos Szots
    @smiklos
    Here again, can someone explain why both HollowConsumer and HollowProducer need a type bound on their Builder class? Basically it's not possible to call these classes without providing an dummy Builder implementation? At least in scala I can't compile this code
    
        val consumer = HollowConsumer
                .newHollowConsumer().build();
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    Hello. Can I get a pair of eyes on this small PR? Netflix/hollow#459
    Mandeep Gandhi
    @welcomemandeep

    Hey folks
    I am getting an NoClassDef error while building my project for one of the dependent classes being used by my data model. Not sure how to get that included. Kindly help
    Execution failed for task ':generateHollowConsumerApi'.

    Lcom/fasterxml/jackson/databind/JsonNode;

    Relevant Stack trace
    Caused by: java.lang.NoClassDefFoundError: Lcom/fasterxml/jackson/databind/JsonNode;
    at com.netflix.hollow.core.write.objectmapper.HollowObjectTypeMapper.<init>(HollowObjectTypeMapper.java:84)
    at com.netflix.hollow.core.write.objectmapper.HollowObjectMapper.getTypeMapper(HollowObjectMapper.java:137)
    at com.netflix.hollow.core.write.objectmapper.HollowObjectMapper.getTypeMapper(HollowObjectMapper.java:109)
    at com.netflix.hollow.core.write.objectmapper.HollowObjectMapper.initializeTypeState(HollowObjectMapper.java:105)
    at com.netflix.nebula.hollow.ApiGeneratorTask.generateApi(ApiGeneratorTask.java:59)
    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:103)

    krishnaram
    @krishnaram

    Hi I am getting this exception

    Please help in this

    Alexandru-Gabriel Gherguț
    @AlexandruGhergut

    if in the current cycle a dataset fails, is there a recommended way for a classic producer to publish the previous state for that dataset? We don't want to fail the cycle and want to publish the successful datasets

    For example, we have

    @HollowPrimaryKey(fields={"orgId"})
    public class DomainFilterConfig {
        private String orgId;
        private List<String> whitelistedClickRedirectDomains;
    }

    and in case of failure, we tried

     stateEngine.getTypeState(failedModel).addAllObjectsFromPreviousCycle()

    but this leads to some strange behavior. The contents of the list changes to ids mixed with strings from other datasets

    Viktor Nyström
    @viqtor
    @AlexandruGhergut the previous is already published no? why wouldn't you want to fail the current cycle? it would lead to a silent error where your dataset never updates but continues to provide new states right?
    Alexandru-Gabriel Gherguț
    @AlexandruGhergut
    @viqtor yes it is published. We don't want to fail the current cycle because other datasets might have been updated. The error would indeed be a silent one but because we cannot publish anything for the failed dataset, the write state will interpret while building the diff that all entries in that dataset have been deleted while we want to keep the ones from the previous cycle. We are not using the incremental producer
    jkade
    @jkade
    I happened to look at the README.md today - is it true that 3.0.1 is the latest "stable" version? https://github.com/Netflix/hollow/blob/master/README.md
    1 reply