Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Bhavesh Mistry
    @bmistry13
    Then we can unwrap the collection and load actual value and then serialize it
    PersistenceUtils unwraps collection into List or Set or Collection
    Thomas Heigl
    @theigl
    @bmistry13 : you can create a custom serializer for Hibernate proxies and persistent collections and unwrap the proxy in the write method
    e.g.
     kryoSerializer = new Kryo() { 
        @Override 
        public Serializer<?> getDefaultSerializer( final Class type ) { 
          if (AbstractPersistentCollection.class.isAssignableFrom( type )) { 
            return new FieldSerializer( kryoSerializer, type ); 
          } 
          return super.getDefaultSerializer( type ); 
        } 
      }
    something along those lines. and then override the write method of the serializer and unwrap the proxy before calling super.write
    payne911
    @payne911
    hi guys
    How does KryoNet work with Java 14's records?
    I'm getting:
    com.esotericsoftware.kryonet.KryoNetException: Error during deserialization.
    Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): ...
    payne911
    @payne911
    Patrik Mihalcin
    @pmihalcin

    Hi all
    I'm dealing with thread safety of Kryo, Input and Output instances. I use com.esotericsoftware:kryo:4.0.2 and Kryo 4 doesn't offer API for pooling of Input and Output instances..

    how do you make Kryo, Input and Output instances thread safe using Kryo 4?

    Patrik Mihalcin
    @pmihalcin

    my take on this is:

    public class KryoSerializer {
    
        private static final KryoFactory factory = () -> {
            Kryo kryo = new Kryo();
            return kryo;
        };
        private static final KryoPool pool = new KryoPool.Builder(factory)
                .softReferences()
                .build();
    
        public byte[] serialize(Object object) {
            Output output = new Output(new ByteArrayOutputStream());
            Kryo kryo = pool.borrow();
            kryo.writeClassAndObject(output, object);
            output.close();
            pool.release(kryo);
            return output.getBuffer();
        }
    
        public Object deserialize(byte[] byteArray) {
            Input input = new Input(byteArray);
            Kryo kryo = pool.borrow();
            Object object = kryo.readClassAndObject(input);
            input.close();
            pool.release(kryo);
            return object;
        }
    }

    Do you see any problem with this implementation?

    Martin Grotzke
    @magro
    @pmihalcin basically this LGTM. IMHO you could/should use try/finally so that the Kryo instance is also released in case of exceptions.
    Following the first option shown in https://github.com/EsotericSoftware/kryo/tree/kryo-4#pooling-kryo-instances, you could pool Input/Output using ThreadLocals. Just make sure to reset them after usage.
    Nicholas Nezis
    @nicknezis
    What is the benefit to creating an ArrayListSerializer like this? Is ArrayList not already covered by the built-in serializers?
    https://github.com/apache/incubator-heron/blob/9d4541b34969fb871754086827544e0c6779c27e/heron/api/src/java/org/apache/heron/api/serializer/KryoSerializer.java#L221
    I'm looking to upgrade this project from 3.0.3 to 5.0.0-RC7, but wasn't sure if these serializers were still needed. If so, what would I need to change to support the newer version of Kryo?
    ┼Żygimantas Gatelis
    @zygisx
    Hi, i have noticed that from version 5 KryoPool class is not included anymore. Couldn't find anything in migration guide either: https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
    Maybe someone could guide me where i can find replacement for it ?
    Thomas Heigl
    @theigl
    @zygisx: Pooling now works slightly differently: https://github.com/EsotericSoftware/kryo#pooling
    We should probably add this to the migration guide
    Vadym Sakhno
    @prineside

    Hi, not sure if I should create a Github issue for it, but here's my problem:

    Kryo 5.0.0 fails to instantiate on Android 11 with this exception:

    java.lang.NoSuchMethodError: No static method of()Ljava/util/Map; in class Ljava/util/Map$-DC; or its super classes
        at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers$JdkImmutableMapSerializer.addDefaultSerializers(ImmutableCollectionsSerializers.java:118)
        at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers.addDefaultSerializers(ImmutableCollectionsSerializers.java:39)
        at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:231)
        ...

    For some reason, there was no problem for it to add default serializers for List.of().
    Not sure what is going on here, somehow it found "java.util.ImmutableCollections" class but failed to call java.util.Map.of(). The device is Pixel 4 with official firmware so I'd expect such behavior on many other devices.

    Is there any way to force Kryo not to load default serializers for immutable collections? Maybe some gradle magic which allows to replace a class or something. I could think of a couple of ways to solve this but all of them would require to change Kryo's source code:

    • Add another Kryo constructor which doesn't load default serializers (or is limited to the primitive ones) - they could be loaded on demand
    • (too dirty) catch exceptions on default serializer instantiation
    • Replace Class.forName() with something else to check if Map.of() and other methods are available (still can't understand why there's no such method)

    Map.of(), List.of(), Set.of() / copyOf() can't be resolved in my ide running on sourceCompatability of Java 8 - I thought it might be the case but still not sure upgrading to Java 9+ makes sense for Android (also would require a bit of .gradle refactoring which I'm not a fan of) so had no chance checking it

    If anyone can explain anything of these to me I'll be very grateful!
    Also thanks for making Kryo, it is cool

    Thomas Heigl
    @theigl
    @prineside: Immutable collection serializers are only added if the class java.util.ImmutableCollections is on the classpath:
    if (isClassAvailable("java.util.ImmutableCollections")) {
        JdkImmutableListSerializer.addDefaultSerializers(kryo);
        JdkImmutableMapSerializer.addDefaultSerializers(kryo);
        JdkImmutableSetSerializer.addDefaultSerializers(kryo);
    }
    does this class exist on android? could you step through the code in ImmutableCollectionsSerializers.addDefaultSerializers() and check why isClassAvailable returns true?
    Thomas Heigl
    @theigl
    according to https://developer.android.com/reference/java/util/Map#immutable, all the Map.of overloads should be available
    we can add a try/catch block or skip adding those serializers for android altogether, but I'd like to understand why this isn't working first
    Thomas Heigl
    @theigl
    it would be great if you could somehow debug this to see what the real problem is.
    Thomas Heigl
    @theigl
    it seems that the static factory methods were added in API version 30: https://developer.android.com/sdk/api_diff/30/changes
    Vadym Sakhno
    @prineside
    Thanks for the tips! I've found the issue - my app was compiled with targetSdkVersion 29. Changed it 30 and Kryo started working like a charm.
    Still don't know why Class.forName("java.util.ImmutableCollections") returned a valid Class
    ykhilaji
    @ykhilaji

    Hello ,
    I have a problem of deserailisation with a complex class case (see log),

    Ps => despite the warning, I noticed nothing as a malfunction,

    I use in my project:
    akka 2.6.11 ( akka cluster , akka streams , akka pubsub )
    scala 2.12,
    play 2.6

    log server:
    [warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
    Serialization trace:
    underlying (play.api.libs.json.JsObject)
    data (models.TaskModel)
    task (actors.websockets.ClientActor$TaskDeleted)
    [info] 2020-11-30 19:09:31,504 - actors.websockets.ClientActor - task deleted, id: 66657
    [warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException

    1 reply
    SharpWriter
    @sharpwriter

    Hello, I am using Kryo 4.0.2 together with "de.javacaffee" % "kryo-serializers" % "0.45" as well as "io.altoo" % "akka-kryo-serialization" % "1.1.0" to do serialization between Akka nodes. Recently we had a case class that had a play.api.libs.json.JsValue as type. JsValue is a trait with various concrete classes for the various parts of a JSON tree (JSString, JSObject, etc...). I noticed that when trying to transfer the case class between nodes, a NullpointerException was raised and the connection between the nodes got disrupted. I only got it to work by explicitly declaring a custom serializer for each concrete implementation of JsValue as indicated below :

    abstract class PlayJsonSerializer[T <: JsValue] extends Serializer[T] {
      override def write(kryo: Kryo, output: Output, `object`: T): Unit = output.writeString(Json.stringify(`object`))
    
      override def read(kryo: Kryo, input: Input, `type`: Class[T]): T = Json.parse(input.readString()).asInstanceOf[T]
    }
    
    class JsValueSerializer extends PlayJsonSerializer[JsValue]
    class JsObjectSerializer extends PlayJsonSerializer[JsObject]
    class JsStringSerializer extends PlayJsonSerializer[JsString]
    class JsNumberSerializer extends PlayJsonSerializer[JsNumber]
    class JsBooleanSerializer extends PlayJsonSerializer[JsBoolean]
    class JsArraySerializer extends PlayJsonSerializer[JsArray]
    class JsNullSerializer extends PlayJsonSerializer[JsNull.type]

    Is this the only way to serialize a Play Json value with Kryo ? Or am I missing something ?

    ykhilaji
    @ykhilaji
    Hi SharpWriter,
    In my case, I translated the json to a conctret value before sending the object into the network, I don't think this is the best way to do it, I would like to implement it your way, you have an example of this solution please?
    SharpWriter
    @sharpwriter
    @ykhilaji : well, the solution is the code snippet above; I'm just adding this to the list of custom kryo serializers. Basically, I'm avoiding the complication with JSValue by simply telling Kryo to serialize to string on the outgoing side, and rehydrating it back to a JsValue on the receiving side.
    pmoncuquet
    @pmoncuquet:matrix.org
    [m]
    hello! i'm trying to use the field @Optional to add a parameter in the object i usally serialized with kryo.
    So i had to deserialized a binary which doesn't have this new field, so i would like set it as optional with a default value. I don't really understand what kind of value is expected in this optional. I use kryo 4.0.2. Thanks
    2 replies
    Dexter Lowe
    @feldoh
    Are there some code samples or a generally preferred option around for consuming a file of many but an unknown number of Kryo objects all of a specific type? I'm streaming objects into the file just by calling kryo.writeObject. On the read side I want to stream through this file completely. Waiting for the underflow doesn't seem good, .available being zero doesn't actually prove there are no bytes left. I'd expect to be able to call ReadObjectOrNull and have it return null if at EOS but I still get the underflow exception. while(input.limit() > input.position() || input.position() == 0) feels a bit odd, especially with the extra start of stream check. Am I missing something blindingly obvious. Something like Files.lines that just gives me a Stream<MyObj> might be nice but Stream support definitely unnecessary.
    7 replies
    Zhaowei Gu
    @zw-g
    hello, I'm a student trying to study Design patterns from KRYO, what are some design patterns, and which class or interface is playing which role in the design pattern?
    3 replies
    Priyam Vora
    @Underrated__bro_twitter
    Hi Can anyone help me with this error? I am getting this error in flink job
    Caused by: java.lang.UnsupportedOperationException: null
    at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
    3 replies
    Vadym Sakhno
    @prineside
    @Underrated__bro_twitter Kryo uses default collection serializer to recreate an object, which involves add() method. Your data is stored in some kind of UnmodifiableCollection which does not allow to use add(). It can be helped by using a custom serializer which should pass all of the collection's data through its constructor. Register and implement your custom serializer for UnmodifiableCollection (or whichever class is used for that problematic field), refer to the Kryo's readme / wiki if you need implementation guideline. Alternatively, search for the existing serializers for your specific data type on github, chances are someone has already done that
    Priyam Vora
    @Underrated__bro_twitter
    Here I am using a Flink job and I have added thee following snipet of code. The EntityEvent class has a unmodifiable collection list which is causing the issue. Is this the right way?
    StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
    Class<?> unmodColl = Class.forName("java.util.Collections$UnmodifiableCollection");
    environment
    .getConfig()
    .addDefaultKryoSerializer(unmodColl, UnmodifiableCollectionsSerializer.class);
    environment.getConfig().addDefaultKryoSerializer(EntityEvent.class, ProtobufSerializer.class);
    Kavitha Durai
    @kavithad:matrix.org
    [m]
    Hello , I am getting below error when executing hive query in EMR
    Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    Serialization trace:
    typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
    reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).
    Caused by: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:706)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
    ... 70 more
    ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
    Kavitha Durai
    @kavithad:matrix.org
    [m]

    :point_up: Edit: Hello , I am getting below error when executing hive query in EMR , sometimes jobs succeeds and it fails intermittently.

    Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
    Serialization trace:
    typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
    colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
    childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
    reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).
    Jonathan D'Orleans
    @jdorleans

    I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

    Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
        at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
        at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
        at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
        at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
        at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

    I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

    Did anyone face similar issue and/or would have a solution for it?

    MikhailPigolkin
    @MikhailPigolkin
    after packing/unpacking, the object is obtained with empty fields. Used kryo.se InstantiatorStrategy(new StdInstantiatorStrategy()); and kryo.se InstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy())); and kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new SerializingInstantiatorStrategy())); the result is an object with empty fields. The object has only two fields long and long. What am I doing wrong?
    2 replies
    Oleh
    @Oleh-M

    Hello everyone
    I'm running my Apache spark 3.3.0 version pipeline on EMR 6.8
    I'm creating my Spark Session with those two options set:

              .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
              .set("spark.kryo.registrationRequired", "true")

    In my ClassPath I only have "io.altoo" %% "akka-kryo-serialization" % "2.1.0" and when I was trying to exclude these:

        ExclusionRule("org.objenesis", "objenesis"),
        ExclusionRule("com.esotericsoftware", "minlog"),
        ExclusionRule("com.esotericsoftware", "reflectasm")

    It doesn't help, so I think this is not an issue
    I keep getting the following error:

    Exception in thread "pool-5-thread-47" java.lang.NoSuchMethodError: com.esotericsoftware.kryo.serializers.FieldSerializer.setIgnoreSyntheticFields(Z)V
    at com.twitter.chill.KryoBase.newDefaultSerializer(KryoBase.scala:67)
    at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:387)
    at com.esotericsoftware.kryo.Kryo.register(Kryo.java:416)
    at org.apache.spark.serializer.KryoSerializer.$anonfun$newKryo$2(KryoSerializer.scala:141)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
    at org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
    at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)

    Could someone please help me where to look for a solution?

    I tried solution described here https://www.jianshu.com/p/fe383e6ce7c7
    but it doesn't help
    David Stancu
    @mach-kernel

    greetings -- i wanted to know how to approach solving for:

    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 4 times, most recent failure: Lost task 0.3 in stage 63.0 (TID 305) (172.16.3.239 executor 4): com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:804)
        at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:307)
        at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:196)

    i'm streaming into spark using the rawinputdstream API -- each frame is [int32 size, (serialized object...)] (what rawSocketStream expects). the serialized object in my case is a byte[]. works fine with the default java serializer. tried preregistering Class.forName("[B") but i think that chill does all of this init. trace logging doesn't show me any objects with ID 13994. does anyone recognize what this object could be?

    David Stancu
    @mach-kernel
    i ended up fixing this since i have control of the publisher -- writing payloads with kryo reify correctly