Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Ghost
    @ghost~5d83f3f9d73408ce4fcb8616
    Hi
    Drew Koszewnik
    @dkoszewnik
    @gazzyt yes, I don't see any changes to the format between these versions.
    mahipal0913
    @mahipal0913
    Hi
    I have question regarding the HashIndex
    sujithramanathan
    @sujithramanathan
    Hi Team,
    Can anyone please answer to my question above.
    Drew Koszewnik
    @dkoszewnik
    @sujithramanathan usually the performance of a HollowProducer can be dramatically optimized by memoizing the POJOs which you add and using the __assigned_ordinal field to inform the HollowObjectMapper that it has already added the record.
    https://hollow.how/data-ingestion/#memoizing-pojos-in-the-hollowobjectmapper
    Try doing this for one or a few record types which have a lot of repetition in your dataset.
    @mahipal0913 sure, what's the question?
    Tim Taylor
    @toolbear

    @sujithramanathan issues (1) and (2) may share a root cause. Drew's reply is a good place to start for issue (1). I will describe issue (2):

    The maximum number of unique records per type is 1<<29 (536,870,912). This because because ordinals are ints, but we reserve 3 bits so we only have 29 bits for the actual ordinal value.

    You mention 520 million unique pairs. This is close enough to the limit that I believe you may be exceeding the ordinal limit (see ghost records in the section on ordinals).

    Two potential workarounds:

    1. reduce the cardinality of your records if possible. How to do this exactly is dependent on the semantics of your dataset, but can be generalized as reduce the precision such that two previously distinct values will now be the same value. One concrete example: replacing float/double in your dataset with an int * a power of 10 (e.g. store 129 instead of 1.2999999523162842 and divide by 100 on the consuming side)

    2. manually "shard" your type into multiple types. Given type A which has high cardinality, replace it with types A1 to A{n}. Use a consistent hash to determine which shard.

    Ultimately if you are at or exceeding the ordinal limit Hollow might not be the right technology for your use case.

    @TinTran64 that sounds like you're constructing the index before the consumer is initialized with data. Are you calling consumer.triggerRefresh()? That or omit withAnnouncementWatcher and call consumer.triggerRefreshTo(1L).
    Tim Taylor
    @toolbear
    @dipsgarg09_twitter Sorry for the late reply. On the consuming end, there is no difference whether the delta came from a full or from an incremental producer. The populated/previous ordinals calls should work the same.
    mahipal0913
    @mahipal0913
    @dkoszewnik We have data with Map<string,string> as one of the property and we would like to do the HashIndex for the Map object
    mahipal0913
    @mahipal0913
    We have created CustomAPIHashIndex customAttrIdx = new CustomAPIHashIndex(consumer,"Node","","attributes.key.value","attributes.value.value");
    customAttrIdx.findNodeMatches("title","tiger001")
    we are able to get the data but we want to apply the index for multiple key values
    like customAttrIdx.findNodeMatches("title","tiger001") & customAttrIdx.findNodeMatches("price","$12.00") get the data in one query
    mahipal0913
    @mahipal0913
    it looks like we can do only one property of the map at a time
    Please let me know if it is not clear I will create example and send it
    mahipal0913
    @mahipal0913

    We have below POJO
    TitlePojo
    {
    private string type:
    private string subType:
    Map<String,String> properties;

    }

    We have data like below

    {
    “type”: “Show”
    “subType”: “comedy”
    “Properties”:
    {
    “title”: “tiger001”
    “groupid”: “001”
    “price”: ”12”
    }
    }

    {
    “type”: “Show”
    “subType”: “comedy”
    “Properties”:
    {
    “title”: “terminator”
    “groupid”: “001”
    “price”: ”12”
    }
    }

    {
    “type”: “Show”
    “subType”: “comedy”
    “Properties”:
    {
    “title”: “Vxtdrop”
    “groupid”: “999”
    “price”: ”18”
    }
    }

    In the above I would like to pull the data with groupid=001 & price=12 in one shot
    DEEPAK GARG
    @dipsgarg09_twitter
    thanks @toolbear. another question : no doubt, there are lot of benefits of hollow over other cache strategies we have in the market. But i am trying to understand , In hollow, we have the complete dataset in memory at a given time but every-time a request comes, we will have to convert the hollow object to our own POJO to use it, wheres in other cache strategies this is not the case, we get our POJO from the cache directly. can you please help me understand how we can optimize or am i misunderstanding ? (AFAIK , it is not advisable to cache hollow objects..)
    sujithramanathan
    @sujithramanathan
    @dkoszewnik @toolbear Thank you. I will check with the points you have mentioned.
    Tim Taylor
    @toolbear

    it is not advisable to cache hollow objects

    Not advisable to use Hollow to populate and cache POJOs on the consuming side. Two broad scenarios in use at Netflix:

    • consumers code against the generated client API that matches the Hollow schema
    • consumers code against Java interfaces with concrete implementations that delegate lookup to the generated client API without caching the results

    In a few cases — types that have low cardinality (like an enum, or list of country codes) which are accessed in a hot path — we create and cache POJOs that are populated and refreshed from the underlying Hollow data. The tradeoff is use more memory to get more performance.

    Tim Taylor
    @toolbear

    @mahipal0913 you would need two different hash indexes

    Side note: there's a type-safe hash index API (undocumented, but supported) called HashIndex. You can see example usage in the unit tests. This works the same as HollowHashIndex but with better API ergonomics.

    mahipal0913
    @mahipal0913
    @toolbear THANKS
    I will look into it and ping you if has more qiestions. Again Thanks
    DEEPAK GARG
    @dipsgarg09_twitter
    @toolbear does hollow provide a pushing mechanism OOTB or do we have to use aws SNS mechanism to listen for updates?
    Currently i am using pull mechanism just like what HollowFilesystemAnnouncementWatcher does polling every 10 sec
    Tim Taylor
    @toolbear
    @dipsgarg09_twitter Hollow doesn't provide a push-based Announcer nor AnnouncementWatcher nor does it have an opinion on push versus polling (beyond providing polling-based examples). You should be able to write an AnnouncementWatcher that receives pushes via SNS.
    Kevin McLarnon
    @kmclarnon
    @toolbear what's the purpose of previousStateRandomizedTag / nextStateRandomizedTag in the hollow blob header? I'm curious because we're doing process level isolation of producers into multiple cloud providers for HA storage and would like consumers to be able to fail over from, say, s3 -> gcs when attempting to read a blob. We control the hollow versioning outside of these producer processes so I expected the binary blobs produced to be identical, but due to this header the failover causes the consumer to fail with "Attempting to apply a delta to a state from which it was not originated!"
    Fuyang Liu
    @liufuyang
    Hi there, can someone help me build a HashIndex on an enum field?
    I have a data class called Promotions, and inside it has a field called status, which is an enum.

    Now I try to make an index like this HashIndex.from(consumer, Promotion.class).usingPath("status.value", String.class) but it does not work.

    Field path "status.value" not found in data set com.netflix.hollow.core.read.engine.HollowReadStateEngine@12219f6a. A schema of type named "Status" does not contain a field for the last segment of the path prefix "status.value".

    Fuyang Liu
    @liufuyang
    When I build an index on a normal string field of the Promotion class, it works. But now I need to select on this enum status and I don't know how should I do it.
    Fuyang Liu
    @liufuyang
    oh, I figured out myself by debugging it, and seems like I can use _name instead of value on it to achieve this.
    hbelliappa
    @hbelliappa
    Hi there,

    I am having difficulty populating the HollowWriteStateEngine with input from a json file.

    hollowProducer.runCycle(newState -> {
    HollowWriteStateEngine stateEngine = newState.getStateEngine();
    HollowJsonAdapter jsonAdapter = new HollowJsonAdapter(stateEngine, "Flights");

    jsonAdapter.populate(json);
    });

    I get the error. Please see below for complete error. Can you point me to what I might be missing

    Caused by: java.lang.NullPointerException: null
    at com.netflix.hollow.jsonadapter.HollowJsonAdapter.parseSubType(HollowJsonAdapter.java:125) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.HollowJsonAdapter.processRecord(HollowJsonAdapter.java:119) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask$1.run(AbstractHollowJsonAdaptorTask.java:92) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    ... 6 common frames omitted

    My data model is

    Airport @PrimaryKey(airportCode){
    string airportCode;
    string airportName;
    string cityName;
    string stateCode;
    string countryCode;
    string cityCode;
    string cityId;
    string areaId;
    string airportMetaphone;
    }

    String {
    string value;
    }


    My Json file is
    {"airportCode":{"value":"ACA"},"airportName":{"value":"Gen Juan N. Álvarez Intl Airport"},"cityName":{"value":"Acapulco"},"stateCode":{"value":"--"},"countryCode":{"value":"MX"},"cityCode":{"value":"ACA"},"cityId":{"value":"2000000234"},"areaId":{"value":null},"airportMetaphone":{"value":"ALFRS"}}


    java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException
    at com.netflix.hollow.api.producer.HollowProducer.runCycle(HollowProducer.java:481) ~[hollow-3.0.1.jar:3.0.1]
    at com.netflix.hollow.api.producer.HollowProducer.runCycle(HollowProducer.java:390) ~[hollow-3.0.1.jar:3.0.1]
    at com.priceline.flights.producerdemo.producer.DemoProducer.produce(DemoProducer.java:54) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
    Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[na:na]
    at com.netflix.hollow.core.util.SimultaneousExecutor.awaitSuccessfulCompletion(SimultaneousExecutor.java:118) ~[hollow-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask.processFile(AbstractHollowJsonAdaptorTask.java:109) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask.processFile(Abstra
    thank you in advance - Harini
    Complete stack trace

    java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException
    at com.netflix.hollow.api.producer.HollowProducer.runCycle(HollowProducer.java:481) ~[hollow-3.0.1.jar:3.0.1]
    at com.netflix.hollow.api.producer.HollowProducer.runCycle(HollowProducer.java:390) ~[hollow-3.0.1.jar:3.0.1]
    at com.priceline.flights.producerdemo.producer.DemoProducer.produce(DemoProducer.java:54) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
    Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.NullPointerException
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[na:na]
    at com.netflix.hollow.core.util.SimultaneousExecutor.awaitSuccessfulCompletion(SimultaneousExecutor.java:118) ~[hollow-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask.processFile(AbstractHollowJsonAdaptorTask.java:109) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask.processFile(AbstractHollowJsonAdaptorTask.java:72) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.HollowJsonAdapter.populate(HollowJsonAdapter.java:102) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.priceline.flights.producerdemo.producer.DemoProducer.lambda$produce$0(DemoProducer.java:58) ~[classes/:na]
    at com.netflix.hollow.api.producer.HollowProducer.runCycle(HollowProducer.java:438) ~[hollow-3.0.1.jar:3.0.1]
    ... 15 common frames omitted
    Caused by: java.lang.RuntimeException: java.lang.NullPointerException
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask$1.run(AbstractHollowJsonAdaptorTask.java:94) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    ... 3 common frames omitted
    Caused by: java.lang.NullPointerException: null
    at com.netflix.hollow.jsonadapter.HollowJsonAdapter.parseSubType(HollowJsonAdapter.java:125) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.HollowJsonAdapter.processRecord(HollowJsonAdapter.java:119) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    at com.netflix.hollow.jsonadapter.AbstractHollowJsonAdaptorTask$1.run(AbstractHollowJsonAdaptorTask.java:92) ~[hollow-jsonadapter-3.0.1.jar:3.0.1]
    ... 6 common frames omitted
    hbelliappa
    @hbelliappa
    the failure is on the json start token { the value of schema.getReferencedType(fieldPosition) is null
    Tim Taylor
    @toolbear

    I think you want the JSON to be "airportCode": "ACA", ... instead of "airportCode": { "value": "ACA"}, ...

    https://hollow.how/data-ingestion/#json-to-hollow

    hbelliappa
    @hbelliappa
    got it
    thx
    it worked!!
    Fuyang Liu
    @liufuyang

    Hey there, is it possible to merge a simple PR like this? @toolbear
    Netflix/hollow#441

    As we are using hollow in our framework, we would like to add some extra metrics in our metric listener class which is a sub class of this consumer AbstractRefreshMetricsListener, so we could override some of the public methods and plugin out own metrics.

    We can already do that for the producer side as there it has no final keyword on those public method on:
    https://github.com/Netflix/hollow/blob/master/hollow/src/main/java/com/netflix/hollow/api/producer/metrics/AbstractProducerMetricsListener.java

    Thus we hope to remove those final keyword on consumer side public methods as well. Thank you :)

    Fuyang Liu
    @liufuyang

    We are also at the same time helping updating the example repo with PRs like this:
    Netflix/hollow-reference-implementation#16

    Would be nice if some of you can give a look? :)

    damianChapman
    @damianChapman
    I have an Test object in my data model that contains two strings corresponding to start and end dates. The value of the start date string is, for example, "2019-10-23T08:12:33.955191013Z"
    Is it possible to create a query, using indexes or otherwise, to select the Test objects that contain a date that is between the start and end dates. If this will not work with strings and the dates need to be longs I can use the number of milliseconds since the epoch instead of strings for the dates. It could be done by getting all of the Test objects in a list and creating a stream with a filter to get the Test objects that contain a date that is between the start and end dates, but I am hoping that there is a more efficient or faster approach. Any help or insights or documentation would be greatly appreciated.
    Fuyang Liu
    @liufuyang
    Hi there, I have a question, on the consumer side, how can I create or new a HString? :)
    As I have a ListOfString and I would like to know whether a String is in it. What could be the efficient way?
    Or can I pass a String type directly into ListOfString.contains(string)?
    hbelliappa
    @hbelliappa

    Can you clarify if Hollow supports java.time.LocalDate or java.util.Date in the schema declaration. When I generate the API from schema the either type there are compilation errors.
    Itinerary @PrimaryKey(id){
    long id;
    string airline;
    string origin;
    string destination;
    Date startDate;
    }
    Date {
    localDate value;
    }

    The generated object is
    public class Date extends HollowObject {

    public Date(DateDelegate delegate, int ordinal) {
        super(delegate, ordinal);
    }
    
    public LocalDate getValue() {
        int refOrdinal = delegate().getValueOrdinal(ordinal);
        if(refOrdinal == -1)
            return null;
        return  **bold api().getLocalDate(refOrdinal); **bold
    }
    
    public FlightsAPI api() {
        return typeApi().getAPI();
    }
    
    public DateTypeAPI typeApi() {
        return delegate().getTypeAPI();
    }
    
    protected DateDelegate delegate() {
        return (DateDelegate)delegate;
    }

    }

    bold api().getLocalDate(refOrdinal); bold - Has compile errors. How do I use Date in Hollow schema?

    angkar
    @angkar
    Hi I am new to Hollow and have a couple of basic questions - we've an architecture where we bring data down from kafka in the cloud to on-prem kubernetes kafka and then use a consumer to push that data to redis - can Hollow replace redis in this case? (if so, where does Hollow store its data?) And, is there any comparison data for Hollow vs redis or memcached?