EsotericSoftware/kryo

Java serialization and cloning: fast, efficient, automatic

Thomas Heigl

@theigl

@zygisx: Pooling now works slightly differently: https://github.com/EsotericSoftware/kryo#pooling

We should probably add this to the migration guide

Vadym Sakhno

@prineside

Hi, not sure if I should create a Github issue for it, but here's my problem:

Kryo 5.0.0 fails to instantiate on Android 11 with this exception:

java.lang.NoSuchMethodError: No static method of()Ljava/util/Map; in class Ljava/util/Map$-DC; or its super classes
    at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers$JdkImmutableMapSerializer.addDefaultSerializers(ImmutableCollectionsSerializers.java:118)
    at com.esotericsoftware.kryo.serializers.ImmutableCollectionsSerializers.addDefaultSerializers(ImmutableCollectionsSerializers.java:39)
    at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:231)
    ...

For some reason, there was no problem for it to add default serializers for List.of().
Not sure what is going on here, somehow it found "java.util.ImmutableCollections" class but failed to call java.util.Map.of(). The device is Pixel 4 with official firmware so I'd expect such behavior on many other devices.

Is there any way to force Kryo not to load default serializers for immutable collections? Maybe some gradle magic which allows to replace a class or something. I could think of a couple of ways to solve this but all of them would require to change Kryo's source code:

Add another Kryo constructor which doesn't load default serializers (or is limited to the primitive ones) - they could be loaded on demand
(too dirty) catch exceptions on default serializer instantiation
Replace Class.forName() with something else to check if Map.of() and other methods are available (still can't understand why there's no such method)

Map.of(), List.of(), Set.of() / copyOf() can't be resolved in my ide running on sourceCompatability of Java 8 - I thought it might be the case but still not sure upgrading to Java 9+ makes sense for Android (also would require a bit of .gradle refactoring which I'm not a fan of) so had no chance checking it

If anyone can explain anything of these to me I'll be very grateful!
Also thanks for making Kryo, it is cool

Thomas Heigl

@theigl

@prineside: Immutable collection serializers are only added if the class java.util.ImmutableCollections is on the classpath:

if (isClassAvailable("java.util.ImmutableCollections")) {
    JdkImmutableListSerializer.addDefaultSerializers(kryo);
    JdkImmutableMapSerializer.addDefaultSerializers(kryo);
    JdkImmutableSetSerializer.addDefaultSerializers(kryo);
}

does this class exist on android? could you step through the code in ImmutableCollectionsSerializers.addDefaultSerializers() and check why isClassAvailable returns true?

Thomas Heigl

@theigl

according to https://developer.android.com/reference/java/util/Map#immutable, all the Map.of overloads should be available

we can add a try/catch block or skip adding those serializers for android altogether, but I'd like to understand why this isn't working first

Thomas Heigl

@theigl

it would be great if you could somehow debug this to see what the real problem is.

Thomas Heigl

@theigl

it seems that the static factory methods were added in API version 30: https://developer.android.com/sdk/api_diff/30/changes

Vadym Sakhno

@prineside

Thanks for the tips! I've found the issue - my app was compiled with targetSdkVersion 29. Changed it 30 and Kryo started working like a charm.
Still don't know why Class.forName("java.util.ImmutableCollections") returned a valid Class

ykhilaji

@ykhilaji

Hello ,
I have a problem of deserailisation with a complex class case (see log),

Ps => despite the warning, I noticed nothing as a malfunction,

I use in my project:
akka 2.6.11 ( akka cluster , akka streams , akka pubsub )
scala 2.12,
play 2.6

log server:
[warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
underlying (play.api.libs.json.JsObject)
data (models.TaskModel)
task (actors.websockets.ClientActor$TaskDeleted)
[info] 2020-11-30 19:09:31,504 - actors.websockets.ClientActor - task deleted, id: 66657
[warn] 2020-11-30 19:09:31,504 - akka.remote.artery.Deserializer - Failed to deserialize message from [akka://application@127.0.0.1:2551] with serializer id [123454323] and manifest []. com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException

1 reply

SharpWriter

@sharpwriter

Hello, I am using Kryo 4.0.2 together with "de.javacaffee" % "kryo-serializers" % "0.45" as well as "io.altoo" % "akka-kryo-serialization" % "1.1.0" to do serialization between Akka nodes. Recently we had a case class that had a play.api.libs.json.JsValue as type. JsValue is a trait with various concrete classes for the various parts of a JSON tree (JSString, JSObject, etc...). I noticed that when trying to transfer the case class between nodes, a NullpointerException was raised and the connection between the nodes got disrupted. I only got it to work by explicitly declaring a custom serializer for each concrete implementation of JsValue as indicated below :

abstract class PlayJsonSerializer[T <: JsValue] extends Serializer[T] {
  override def write(kryo: Kryo, output: Output, `object`: T): Unit = output.writeString(Json.stringify(`object`))

  override def read(kryo: Kryo, input: Input, `type`: Class[T]): T = Json.parse(input.readString()).asInstanceOf[T]
}

class JsValueSerializer extends PlayJsonSerializer[JsValue]
class JsObjectSerializer extends PlayJsonSerializer[JsObject]
class JsStringSerializer extends PlayJsonSerializer[JsString]
class JsNumberSerializer extends PlayJsonSerializer[JsNumber]
class JsBooleanSerializer extends PlayJsonSerializer[JsBoolean]
class JsArraySerializer extends PlayJsonSerializer[JsArray]
class JsNullSerializer extends PlayJsonSerializer[JsNull.type]

Is this the only way to serialize a Play Json value with Kryo ? Or am I missing something ?

ykhilaji

@ykhilaji

Hi SharpWriter,
In my case, I translated the json to a conctret value before sending the object into the network, I don't think this is the best way to do it, I would like to implement it your way, you have an example of this solution please?

SharpWriter

@sharpwriter

@ykhilaji : well, the solution is the code snippet above; I'm just adding this to the list of custom kryo serializers. Basically, I'm avoiding the complication with JSValue by simply telling Kryo to serialize to string on the outgoing side, and rehydrating it back to a JsValue on the receiving side.

pmoncuquet

@pmoncuquet:matrix.org

[m]

hello! i'm trying to use the field @Optional to add a parameter in the object i usally serialized with kryo.
So i had to deserialized a binary which doesn't have this new field, so i would like set it as optional with a default value. I don't really understand what kind of value is expected in this optional. I use kryo 4.0.2. Thanks

2 replies

Dexter Lowe

@feldoh

Are there some code samples or a generally preferred option around for consuming a file of many but an unknown number of Kryo objects all of a specific type? I'm streaming objects into the file just by calling kryo.writeObject. On the read side I want to stream through this file completely. Waiting for the underflow doesn't seem good, .available being zero doesn't actually prove there are no bytes left. I'd expect to be able to call ReadObjectOrNull and have it return null if at EOS but I still get the underflow exception. while(input.limit() > input.position() || input.position() == 0) feels a bit odd, especially with the extra start of stream check. Am I missing something blindingly obvious. Something like Files.lines that just gives me a Stream<MyObj> might be nice but Stream support definitely unnecessary.

7 replies

Zhaowei Gu

@zw-g

hello, I'm a student trying to study Design patterns from KRYO, what are some design patterns, and which class or interface is playing which role in the design pattern?

3 replies

Priyam Vora

@Underrated__bro_twitter

Hi Can anyone help me with this error? I am getting this error in flink job
Caused by: java.lang.UnsupportedOperationException: null
at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)

3 replies

Vadym Sakhno

@prineside

@Underrated__bro_twitter Kryo uses default collection serializer to recreate an object, which involves add() method. Your data is stored in some kind of UnmodifiableCollection which does not allow to use add(). It can be helped by using a custom serializer which should pass all of the collection's data through its constructor. Register and implement your custom serializer for UnmodifiableCollection (or whichever class is used for that problematic field), refer to the Kryo's readme / wiki if you need implementation guideline. Alternatively, search for the existing serializers for your specific data type on github, chances are someone has already done that

Priyam Vora

@Underrated__bro_twitter

Here I am using a Flink job and I have added thee following snipet of code. The EntityEvent class has a unmodifiable collection list which is causing the issue. Is this the right way?
StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();
Class<?> unmodColl = Class.forName("java.util.Collections$UnmodifiableCollection");
environment
.getConfig()
.addDefaultKryoSerializer(unmodColl, UnmodifiableCollectionsSerializer.class);
environment.getConfig().addDefaultKryoSerializer(EntityEvent.class, ProtobufSerializer.class);

Kavitha Durai

@kavithad:matrix.org

[m]

Hello , I am getting below error when executing hive query in EMR
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
Serialization trace:
typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).

Caused by: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:706)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:214)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
... 70 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :

Kavitha Durai

@kavithad:matrix.org

[m]

:point_up: Edit: Hello , I am getting below error when executing hive query in EMR , sometimes jobs succeeds and it fails intermittently.

Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 109, Size: 98
Serialization trace:
typeInfo (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.FilterOperator)
childOperators (org.apache.hadoop.hive.ql.exec.PTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:176).

Jonathan D'Orleans

@jdorleans

I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
    at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
    at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
    at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
    at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
    at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

Did anyone face similar issue and/or would have a solution for it?

MikhailPigolkin

@MikhailPigolkin

after packing/unpacking, the object is obtained with empty fields. Used kryo.se InstantiatorStrategy(new StdInstantiatorStrategy()); and kryo.se InstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy())); and kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new SerializingInstantiatorStrategy())); the result is an object with empty fields. The object has only two fields long and long. What am I doing wrong?

2 replies

Oleh

@Oleh-M

Hello everyone
I'm running my Apache spark 3.3.0 version pipeline on EMR 6.8
I'm creating my Spark Session with those two options set:

          .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
          .set("spark.kryo.registrationRequired", "true")

In my ClassPath I only have "io.altoo" %% "akka-kryo-serialization" % "2.1.0" and when I was trying to exclude these:

    ExclusionRule("org.objenesis", "objenesis"),
    ExclusionRule("com.esotericsoftware", "minlog"),
    ExclusionRule("com.esotericsoftware", "reflectasm")

It doesn't help, so I think this is not an issue
I keep getting the following error:

Exception in thread "pool-5-thread-47" java.lang.NoSuchMethodError: com.esotericsoftware.kryo.serializers.FieldSerializer.setIgnoreSyntheticFields(Z)V
at com.twitter.chill.KryoBase.newDefaultSerializer(KryoBase.scala:67)
at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:387)
at com.esotericsoftware.kryo.Kryo.register(Kryo.java:416)
at org.apache.spark.serializer.KryoSerializer.$anonfun$newKryo$2(KryoSerializer.scala:141)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
at org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)

Could someone please help me where to look for a solution?

I tried solution described here https://www.jianshu.com/p/fe383e6ce7c7
but it doesn't help

David Stancu

@mach-kernel

greetings -- i wanted to know how to approach solving for:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 4 times, most recent failure: Lost task 0.3 in stage 63.0 (TID 305) (172.16.3.239 executor 4): com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
    at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:804)
    at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:307)
    at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:196)

i'm streaming into spark using the rawinputdstream API -- each frame is [int32 size, (serialized object...)] (what rawSocketStream expects). the serialized object in my case is a byte[]. works fine with the default java serializer. tried preregistering Class.forName("[B") but i think that chill does all of this init. trace logging doesn't show me any objects with ID 13994. does anyone recognize what this object could be?

David Stancu

@mach-kernel

i ended up fixing this since i have control of the publisher -- writing payloads with kryo reify correctly

Where communities thrive

People

Repo info

Activity

:point_up: Edit: Hello , I am getting below error when executing hive query in EMR , sometimes jobs succeeds and it fails intermittently.