Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Bhavesh Mistry
    @bmistry13
    To prevent lazy loading I am initializing
    But I wanted to know how to create custom serializer that just clean Proxy and use exiting serializer to serialize object
    I had to hack ObjectField
            /**** FIX FOR HIBERNATE OBJECT */
            if (PersistenceUtils.isProxy(value)){
                if(!Hibernate.isInitialized(value)){
                    if (INFO)
                        info("kryo", "uninitialized hibernate proxy detected  and initializing: " + this + " (" + object.getClass().getName() + ")" + " pos=" + output.position());
                    Hibernate.initialize(value);
                }
                value = PersistenceUtils.cleanFromProxies(value);
                if (INFO)
                    info("kryo", "Write field detected hibernate proxy and unproxying: " + this + " (" + object.getClass().getName() + ")" + " pos=" + output.position());
            }
    
            /****END  OF FIX *****/
    So I do this outside of ObjectField class
    I would appreciate any help or pointer
    Bhavesh Mistry
    @bmistry13
    @magro It same exact issue EsotericSoftware/kryo#580
    @magro "can't you write a serializer to handle this? The serializer could detect if it's a proxy object, get the implementation, and hand it off to another serializer" Do you have an example of wrapper serialization which will effectively do above and call other serializer?
    Each of Hibernate Entity is HibernateProxy or PersistentCollection
    Then we can unwrap the collection and load actual value and then serialize it
    Bhavesh Mistry
    @bmistry13
    PersistenceUtils unwraps collection into List or Set or Collection
    Thomas Heigl
    @theigl
    @bmistry13 : you can create a custom serializer for Hibernate proxies and persistent collections and unwrap the proxy in the write method
    e.g.
     kryoSerializer = new Kryo() { 
        @Override 
        public Serializer<?> getDefaultSerializer( final Class type ) { 
          if (AbstractPersistentCollection.class.isAssignableFrom( type )) { 
            return new FieldSerializer( kryoSerializer, type ); 
          } 
          return super.getDefaultSerializer( type ); 
        } 
      }
    something along those lines. and then override the write method of the serializer and unwrap the proxy before calling super.write
    payne911
    @payne911
    hi guys
    How does KryoNet work with Java 14's records?
    I'm getting:
    com.esotericsoftware.kryonet.KryoNetException: Error during deserialization.
    Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): ...
    payne911
    @payne911
    Patrik Mihalcin
    @pmihalcin

    Hi all
    I'm dealing with thread safety of Kryo, Input and Output instances. I use com.esotericsoftware:kryo:4.0.2 and Kryo 4 doesn't offer API for pooling of Input and Output instances..

    how do you make Kryo, Input and Output instances thread safe using Kryo 4?

    Patrik Mihalcin
    @pmihalcin

    my take on this is:

    public class KryoSerializer {
    
        private static final KryoFactory factory = () -> {
            Kryo kryo = new Kryo();
            return kryo;
        };
        private static final KryoPool pool = new KryoPool.Builder(factory)
                .softReferences()
                .build();
    
        public byte[] serialize(Object object) {
            Output output = new Output(new ByteArrayOutputStream());
            Kryo kryo = pool.borrow();
            kryo.writeClassAndObject(output, object);
            output.close();
            pool.release(kryo);
            return output.getBuffer();
        }
    
        public Object deserialize(byte[] byteArray) {
            Input input = new Input(byteArray);
            Kryo kryo = pool.borrow();
            Object object = kryo.readClassAndObject(input);
            input.close();
            pool.release(kryo);
            return object;
        }
    }

    Do you see any problem with this implementation?

    Martin Grotzke
    @magro
    @pmihalcin basically this LGTM. IMHO you could/should use try/finally so that the Kryo instance is also released in case of exceptions.
    Following the first option shown in https://github.com/EsotericSoftware/kryo/tree/kryo-4#pooling-kryo-instances, you could pool Input/Output using ThreadLocals. Just make sure to reset them after usage.
    Nicholas Nezis
    @nicknezis
    What is the benefit to creating an ArrayListSerializer like this? Is ArrayList not already covered by the built-in serializers?
    https://github.com/apache/incubator-heron/blob/9d4541b34969fb871754086827544e0c6779c27e/heron/api/src/java/org/apache/heron/api/serializer/KryoSerializer.java#L221
    I'm looking to upgrade this project from 3.0.3 to 5.0.0-RC7, but wasn't sure if these serializers were still needed. If so, what would I need to change to support the newer version of Kryo?
    ┼Żygimantas Gatelis
    @zygisx
    Hi, i have noticed that from version 5 KryoPool class is not included anymore. Couldn't find anything in migration guide either: https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
    Maybe someone could guide me where i can find replacement for it ?
    Thomas Heigl
    @theigl
    @zygisx: Pooling now works slightly differently: https://github.com/EsotericSoftware/kryo#pooling
    We should probably add this to the migration guide
    Vadym Sakhno
    @prineside

    Hi, not sure if I should create a Github issue for it, but here's my problem:

    Kryo 5.0.0 fails to instantiate on Android 11 with this exception:

    java.lang.NoSuchMethodError: No static method of()Ljava/util/Map; in class Ljava/util/Map$-DC; or its super classes
        at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers$JdkImmutableMapSerializer.addDefaultSerializers(ImmutableCollectionsSerializers.java:118)
        at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers.addDefaultSerializers(ImmutableCollectionsSerializers.java:39)
        at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:231)
        ...

    For some reason, there was no problem for it to add default serializers for List.of().
    Not sure what is going on here, somehow it found "java.util.ImmutableCollections" class but failed to call java.util.Map.of(). The device is Pixel 4 with official firmware so I'd expect such behavior on many other devices.

    Is there any way to force Kryo not to load default serializers for immutable collections? Maybe some gradle magic which allows to replace a class or something. I could think of a couple of ways to solve this but all of them would require to change Kryo's source code:

    • Add another Kryo constructor which doesn't load default serializers (or is limited to the primitive ones) - they could be loaded on demand
    • (too dirty) catch exceptions on default serializer instantiation
    • Replace Class.forName() with something else to check if Map.of() and other methods are available (still can't understand why there's no such method)

    Map.of(), List.of(), Set.of() / copyOf() can't be resolved in my ide running on sourceCompatability of Java 8 - I thought it might be the case but still not sure upgrading to Java 9+ makes sense for Android (also would require a bit of .gradle refactoring which I'm not a fan of) so had no chance checking it

    If anyone can explain anything of these to me I'll be very grateful!
    Also thanks for making Kryo, it is cool

    Thomas Heigl
    @theigl
    @prineside: Immutable collection serializers are only added if the class java.util.ImmutableCollections is on the classpath:
    if (isClassAvailable("java.util.ImmutableCollections")) {
        JdkImmutableListSerializer.addDefaultSerializers(kryo);
        JdkImmutableMapSerializer.addDefaultSerializers(kryo);
        JdkImmutableSetSerializer.addDefaultSerializers(kryo);
    }
    does this class exist on android? could you step through the code in ImmutableCollectionsSerializers.addDefaultSerializers() and check why isClassAvailable returns true?
    Thomas Heigl
    @theigl
    according to https://developer.android.com/reference/java/util/Map#immutable, all the Map.of overloads should be available
    we can add a try/catch block or skip adding those serializers for android altogether, but I'd like to understand why this isn't working first
    Thomas Heigl
    @theigl
    it would be great if you could somehow debug this to see what the real problem is.
    Thomas Heigl
    @theigl
    it seems that the static factory methods were added in API version 30: https://developer.android.com/sdk/api_diff/30/changes
    Vadym Sakhno
    @prineside
    Thanks for the tips! I've found the issue - my app was compiled with targetSdkVersion 29. Changed it 30 and Kryo started working like a charm.
    Still don't know why Class.forName("java.util.ImmutableCollections") returned a valid Class
    ykhilaji
    @ykhilaji

    Hello ,
    I have a problem of deserailisation with a complex class case (see log),

    Ps => despite the warning, I noticed nothing as a malfunction,

    I use in my project:
    akka 2.6.11 ( akka cluster , akka streams , akka pubsub )
    scala 2.12,
    play 2.6

    log server:
    [warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
    Serialization trace:
    underlying (play.api.libs.json.JsObject)
    data (models.TaskModel)
    task (actors.websockets.ClientActor$TaskDeleted)
    [info] 2020-11-30 19:09:31,504 - actors.websockets.ClientActor - task deleted, id: 66657
    [warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException

    1 reply
    SharpWriter
    @sharpwriter

    Hello, I am using Kryo 4.0.2 together with "de.javacaffee" % "kryo-serializers" % "0.45" as well as "io.altoo" % "akka-kryo-serialization" % "1.1.0" to do serialization between Akka nodes. Recently we had a case class that had a play.api.libs.json.JsValue as type. JsValue is a trait with various concrete classes for the various parts of a JSON tree (JSString, JSObject, etc...). I noticed that when trying to transfer the case class between nodes, a NullpointerException was raised and the connection between the nodes got disrupted. I only got it to work by explicitly declaring a custom serializer for each concrete implementation of JsValue as indicated below :

    abstract class PlayJsonSerializer[T <: JsValue] extends Serializer[T] {
      override def write(kryo: Kryo, output: Output, `object`: T): Unit = output.writeString(Json.stringify(`object`))
    
      override def read(kryo: Kryo, input: Input, `type`: Class[T]): T = Json.parse(input.readString()).asInstanceOf[T]
    }
    
    class JsValueSerializer extends PlayJsonSerializer[JsValue]
    class JsObjectSerializer extends PlayJsonSerializer[JsObject]
    class JsStringSerializer extends PlayJsonSerializer[JsString]
    class JsNumberSerializer extends PlayJsonSerializer[JsNumber]
    class JsBooleanSerializer extends PlayJsonSerializer[JsBoolean]
    class JsArraySerializer extends PlayJsonSerializer[JsArray]
    class JsNullSerializer extends PlayJsonSerializer[JsNull.type]

    Is this the only way to serialize a Play Json value with Kryo ? Or am I missing something ?

    ykhilaji
    @ykhilaji
    Hi SharpWriter,
    In my case, I translated the json to a conctret value before sending the object into the network, I don't think this is the best way to do it, I would like to implement it your way, you have an example of this solution please?
    SharpWriter
    @sharpwriter
    @ykhilaji : well, the solution is the code snippet above; I'm just adding this to the list of custom kryo serializers. Basically, I'm avoiding the complication with JSValue by simply telling Kryo to serialize to string on the outgoing side, and rehydrating it back to a JsValue on the receiving side.
    pmoncuquet
    @pmoncuquet:matrix.org
    [m]
    hello! i'm trying to use the field @Optional to add a parameter in the object i usally serialized with kryo.
    So i had to deserialized a binary which doesn't have this new field, so i would like set it as optional with a default value. I don't really understand what kind of value is expected in this optional. I use kryo 4.0.2. Thanks
    2 replies
    Dexter Lowe
    @feldoh
    Are there some code samples or a generally preferred option around for consuming a file of many but an unknown number of Kryo objects all of a specific type? I'm streaming objects into the file just by calling kryo.writeObject. On the read side I want to stream through this file completely. Waiting for the underflow doesn't seem good, .available being zero doesn't actually prove there are no bytes left. I'd expect to be able to call ReadObjectOrNull and have it return null if at EOS but I still get the underflow exception. while(input.limit() > input.position() || input.position() == 0) feels a bit odd, especially with the extra start of stream check. Am I missing something blindingly obvious. Something like Files.lines that just gives me a Stream<MyObj> might be nice but Stream support definitely unnecessary.
    7 replies
    Zhaowei Gu
    @zw-g
    hello, I'm a student trying to study Design patterns from KRYO, what are some design patterns, and which class or interface is playing which role in the design pattern?
    3 replies
    Priyam Vora
    @Underrated__bro_twitter
    Hi Can anyone help me with this error? I am getting this error in flink job
    Caused by: java.lang.UnsupportedOperationException: null
    at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
    3 replies
    Vadym Sakhno
    @prineside
    @Underrated__bro_twitter Kryo uses default collection serializer to recreate an object, which involves add() method. Your data is stored in some kind of UnmodifiableCollection which does not allow to use add(). It can be helped by using a custom serializer which should pass all of the collection's data through its constructor. Register and implement your custom serializer for UnmodifiableCollection (or whichever class is used for that problematic field), refer to the Kryo's readme / wiki if you need implementation guideline. Alternatively, search for the existing serializers for your specific data type on github, chances are someone has already done that
    Priyam Vora
    @Underrated__bro_twitter
    Here I am using a Flink job and I have added thee following snipet of code. The EntityEvent class has a unmodifiable collection list which is causing the issue. Is this the right way?
    StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
    Class<?> unmodColl = Class.forName("java.util.Collections$UnmodifiableCollection");
    environment
    .getConfig()
    .addDefaultKryoSerializer(unmodColl, UnmodifiableCollectionsSerializer.class);
    environment.getConfig().addDefaultKryoSerializer(EntityEvent.class, ProtobufSerializer.class);
    Kavitha Durai
    @kavithad:matrix.org
    [m]
    Hello , I am getting below error when executing hive query in EMR
    Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    Serialization trace:
    typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
    reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).
    Caused by: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:706)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
    ... 70 more
    ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
    Kavitha Durai
    @kavithad:matrix.org
    [m]

    :point_up: Edit: Hello , I am getting below error when executing hive query in EMR , sometimes jobs succeeds and it fails intermittently.

    Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    Serialization trace:
    typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
    reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).
    Jonathan D'Orleans
    @jdorleans

    I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

    Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
        at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
        at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
        at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
        at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
        at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

    I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

    Did anyone face similar issue and/or would have a solution for it?

    MikhailPigolkin
    @MikhailPigolkin
    after packing/unpacking, the object is obtained with empty fields. Used kryo.se InstantiatorStrategy(new StdInstantiatorStrategy()); and kryo.se InstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy())); and kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new SerializingInstantiatorStrategy())); the result is an object with empty fields. The object has only two fields long and long. What am I doing wrong?
    2 replies