Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Martin Grotzke
@magro
Hi guys, we're looking for feedback regarding the default setting of Kryo.registrationRequired, would love to hear your thoughts here: https://groups.google.com/d/msg/kryo-users/DaUaUayYAJs/xXNgv9D_BAAJ
Ian O'Connell
@ianoc
Replied to the thread
Ian O'Connell
@ianoc
@magro thanks for reaching out for feedback btw. Reply sort of short was trying to squeeze it in between meetings in work. appreciate the looking to see what everyone thinks!
Martin Grotzke
@magro
@ianoc thanks for responding, definitely helps!
Inash Zubair
@inash
Hi. I don't see Scala ListMap in the list of supported collection types. Are there plans of supporting it and other types? I'm currently using chill-akka serializer for akka remoting, and ListMaps appear out of order on the other side. However, I could always recreate an ordered ListMap using the values .
P. Oscar Boykin
@johnynek
@inash I don't think we added it since it is more rare. Happy to take a PR to add it.
Inash Zubair
@inash
@johnynek Seems I was wrong. Was using mutable.LinkedHashMap, but after converting to an immutable.ListMap it was working fine and the order was maintained.
Anton Sviridov
@keynmol

Hey guys. Is there any log of deserialisation regressions between 0.6.0 and 0.8.0? I recently upgraded and noticed serious performance regression.

The file is 4.5M, basically a serialised structure case class ContextTokenStore(tokenLookup: Map[String, Int], idLookup: Map[Int, String], tokenTotalCounts: Map[Int, Int]).

Kryo is initialised as such:

val instantiator = (new ScalaKryoInstantiator).setReferences(false)
  val kryo: KryoPool = KryoPool.withByteArrayOutputStream(10, instantiator)
...
def load[T](file: File): T = {
    val bytes = file.loadBytes
    val store: T = kryo.fromBytes(bytes).asInstanceOf[T]

    store
  }

The file size is the same if produced by both 0.6.0 and 0.8.0, but the loading from disk times vary significantly (basically loading 20 times, times in milliseconds):

0.6.0

[loading] took 562, and then 228, 179, 168, 117, 120, 138, 128, 125, 139, 130, 123, 137, 134, 120, 132, 141, 143, 144, 154

0.8.0

 [loading] took 3081, and then 1859, 1813, 1722, 1622, 1688, 1704, 1881, 2603, 2164, 2040, 1946, 2010, 2063, 2053, 2159, 2717, 2065, 2276, 2381

Any clue as to where to look?

Ian O'Connell
@ianoc
@keynmol I think the main thing here would be that 0.8.0 uses a newer kryo, the 0.3.x rather than 0.2.x . I'm not exactly sure what there though -- https://github.com/EsotericSoftware/kryo/blob/master/CHANGES.md#2240---300-2014-10-04 is probably a good place to start.
Does it matter what your type T is?
Anton Sviridov
@keynmol
It seems to have slowed down for all the kryo-based types that we use (they're all pretty big, in production they're hundreds of megabytes). I'm gonna try an whip up a simple project to reproduce this
Ian O'Connell
@ianoc
Yeah that would be great thanks. The way we use Chill inside MR jobs/scalding mostly its often unfortunately relatively opaque to measure in general. So a benchmark with regression numbers like the above would be great
Anton Sviridov
@keynmol
Argh. Can't reproduce it in an isolated environment. Reproduced it many times within the project's classpath. One thing I noticed when using 0.8.0 is that self-time on string looks awful:
whilst its counterpart in earlier kryo 2.21 (I guess it's readString?) almost doesn't show up in profiling at all
Ian O'Connell
@ianoc
By the looks of the kryo code -- I think that method is basically a fancy toString @ https://github.com/EsotericSoftware/kryo/blob/kryo-parent-3.0.3/src/com/esotericsoftware/kryo/util/Util.java#L104
So I'd suspect something is odd if its showing up at all
Oh -- could this be the log level of your app -- on the off chance if you touched off https://github.com/EsotericSoftware/kryo/blob/kryo-parent-3.0.3/src/com/esotericsoftware/kryo/util/Util.java#L88 it would be problematic
Anton Sviridov
@keynmol
huh when I enabled debug level in log4j2 I saw every single object being logged
But I haven't changed any configuration for kryo logging
that's a very good catch - do you know why exactly would kryo log anything?
sheeeeit
Anton Sviridov
@keynmol
@ianoc you're a star
Ian O'Connell
@ianoc
config setting fix it?
Anton Sviridov
@keynmol

it's a bit more involved

We had an exclusion rule:

libraryDependencies ~= { _.map(_.excludeAll(
      ExclusionRule("com.esotericsoftware", "minlog"),
      ExclusionRule("commons-logging"),
      ExclusionRule("log4j", "log4j"),
      ExclusionRule("org.slf4j")
    )) },

and a merge strategy

      // com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
      // com/esotericsoftware/minlog/minlog/1.2-slf4j-jdanbrown-0/minlog-1.2-slf4j-jdanbrown-0.jar
      case PathList("com", "esotericsoftware", "minlog", xs @ _*) =>
        MergeStrategy.last

So in the dependencyTree of the branch with updated chill there was only +-com.esotericsoftware.minlog:minlog:1.2-slf4j-jdanbrown-0, but the old branch without regressions had

[info]   | +-com.esotericsoftware.minlog:minlog:1.2
[info]   +-com.esotericsoftware.minlog:minlog:1.2
[info]   +-com.esotericsoftware.minlog:minlog:1.2-slf4j-jdanbrown-0 (evicted by: 1.2)
[info]   | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   |     +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |     +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | |     +-com.esotericsoftware.minlog:minlog:1.2

I'm not sure how minlog works, but removing exclusion rule made things even a bit faster than old branch:

[loading] took 419, and then 152, 126, 133, 144, 141, 149, 205, 161, 166, 176, 209, 154, 160, 154, 165, 165, 149, 147, 172
So I guess the lesson here is kicking out com.esotericsoftware.minlog:minlog:1.2-slf4j-jdanbrown-0 wherever the hell it comes. @ianoc thank you so much :)
Ian O'Connell
@ianoc
Sweet! So at a guess i think what might have happened is that by getting a different logger on the classpath the debug code didn't get elminiated as part of optimization. I think it needs to get dumped out by realizing it won't be used.
Logging and classpaths, so terrible on the JVM
No problem, glad your sorted!
(re-writing the minlog interfaces i think in that jar would enable this behavior)
Anton Sviridov
@keynmol
jesus. tests passed, assembly passed, everything passed. it's a miracle I spotted this regression before shipping this. Thanks, JVMobama
Ian O'Connell
@ianoc
ha, yeah, good spot. Thats a rough one to do. Unfortunately unless you shade things you'll probably need to be careful downstream to ensure nothing can drag in the bad dep and cause the regression if you consume it in another project
Anton Sviridov
@keynmol
yeah, happens all the time with unshaded transitive dependencies as well. but I've never seen it backfire so gorgeously in runtime
Ian O'Connell
@ianoc
Yeah, usually it explodes at runtime, this feels worse
rimeh bennjima
@rimeh-bennjima
Hallo I want to deserialize a data but some of my data is deserialize correctly and for some other data i have an Exception
I used Kryo-sharded-4.0.1
twitter/chill#311
Wouter Zorgdrager
@wzorgdrager

Does the Externalizer also work with generics & implicits? i.e.

class AvroSerde[T : ClassTag : TypeTag](implicit eSchema: Externalizer[SchemaFor[T]], 
                                    eFrom: Externalizer[FromRecord[T]], 
                                    eTo: Externalizer[ToRecord[T]]) extends AbstractSerde[T]

Will this work?

Jacek Kunicki
@rucek
Hi everyone! When used with Akka, does chill allow to use a different Kryo serializer (like CompatibleFieldSerializer) instead of the default one (FieldSerializer)? I was looking at the instructions about implementing a custom AkkaSerializer, but couldn’t find a way to customize the serializer there.
Rodrigo
@rodrigotn
Hi all, Is there a version for scala 2.13?