suzaku-io/boopickle

Binary serialization library for efficient network communication

Andriy Plokhotnyuk

@plokhotnyuk

Is Any for scala.Any?

jhegedus42

@jhegedus42

hmmm

good question

let's say

more like Map[String,ReferencedValue[_]]

I am not sure what is the difference betwen the two

but basically

actuall

yes

but you know what

i think i just store Map[String,JSON]

and then problem is solved :)

and use circe

Andriy Plokhotnyuk

@plokhotnyuk

Is it required to work with scala.js?

jhegedus42

@jhegedus42

will

well

yeah

ideally :)

Otto Chrons

@ochrons

same as all other serialization libraries, it needs to know the types at compile time and you need to provide picklers for them

Dermot Haughey

@hderms

anyone have any idea how much faster boopickle would be than circe?

especially if boopickle was using binary codecs

we are trying to serialize like 200MB of data

Andriy Plokhotnyuk

@plokhotnyuk

it depends a lot on payload and environment... as an example jsoniter-scala can serialize with speed up to 3Gb/sec... please see the full picture here

Dermot Haughey

@hderms

@plokhotnyuk thanks I'll give it a look

Otto Chrons

@ochrons

for text, Boopickle is not faster, but for something like numbers, especially floating point, it can be much much faster than any JSON codec

Boopickle comes with a performance test suite that you can easily modify to match your data and see the differences

Nils Kilden-Pedersen

@nilskp

@ochrons I'm surprised you write that, as JSON text needs to be escaped, unlike (presumably) Boopickle. At the very least that would lead to per-char checking, which might be cheap, but not free, and any object field will have a String name, which doesn't even exist in a binary codec.
Am I missing something?

Otto Chrons

@ochrons

not sure what you're asking :)

Nils Kilden-Pedersen

@nilskp

@ochrons I'm confused as to why you'd write that "for text, Boopickle is not faster".

It should be faster, since JSON de/serialization requires escaping, which Boopickle, being binary, presumably does not.

Otto Chrons

@ochrons

maybe on JVM the difference is not that big, but on JS the platform has highly optimized JSON.stringify function which handles strings quite much faster than doing UTF8ish encoding in JS code

Nils Kilden-Pedersen

@nilskp

So what you're saying is, that when serializing to a JS environment, Boopickle is not faster for text, because the Boopickle's JS implementation of String encoding is slower than JSON.stringify?

Otto Chrons

@ochrons

yeah, and I'm not too sure about JVM perf either, would need to be benchmarked

at least on JVM the string encoding perf is on the same level with Google protobuf :)

Nils Kilden-Pedersen

@nilskp

my point was re. JVM <-> JVM communication, and I would expect Boopickle to be faster than any JVM <-> JVM JSON implementation, because no escaping is needed.

Otto Chrons

@ochrons

could be faster, although typically not much needs to be escaped in JSON strings

Nils Kilden-Pedersen

@nilskp

Correct, but every single char still needs to be range/char checked.

might be fast, but cannot be free.

Otto Chrons

@ochrons

true, the perf statements are mostly taken in the JS context, as that was the driving factor when I was developing Boopickle

Nils Kilden-Pedersen

@nilskp

gotcha. Thanks for clarifying.

Otto Chrons

@ochrons

:+1:

moritz bust

@busti

I am trying to pickle a custom collection type using a transformPickler.
The collection is a strictly alternating sequence of two types and is implemented like this: class Alternating[Even, Odd](val first: Even, val rest: Seq[(Odd, Even)])
My Pickling code looks like this:

  implicit def alternatingPickler[Even, Odd] =
    transformPickler[Alternating[Even, Odd], (Even, Seq[(Odd, Even)])] {
      case (first, rest) => Alternating(first, rest)
    } (a => (a.first, a.rest))

I am getting an error message when compiling:

[error] C:\Users\mbust\Documents\projects\delta-t\modules\core\shared\src\main\scala\io\delta_t\collection\serializers\Boopickle.scala:14:7: exception during macro expansion:
[error] java.lang.RuntimeException: Enclosure: source-C:\Users\mbust\Documents\projects\delta-t\modules\core\shared\src\main\scala\io\delta_t\collection\serializers\Boopickle.scala,line-14,offset=432, type = Even
[error]         at boopickle.PicklerMaterializersImpl$.materializePickler(PicklerMaterializersImpl.scala:106)
[error]     } (a => (a.first, a.rest))
[error]       ^

Does transformPickler support generics? Or is there something else I am doing wrong?

Otto Chrons

@ochrons

you need to provide implicit picklers for Even and Odd

moritz bust

@busti

Like this implicit def alternatingPickler[Even : Pickler, Odd : Pickler]?

Otto Chrons

@ochrons

yea

moritz bust

@busti

Neat, thanks :)

moritz bust

@busti

The interface of transformPickler compared to compositePickler.addTransform is a bit odd.
In the former it is (transformFrom)(transformTo) and in the latter it is (transformTo, transformFrom).
Is there a reason for that?

Otto Chrons

@ochrons

the curried version allows using the types from the first argument in the second argument

with addTransform that is not needed IIRC

Where communities thrive

People

Repo info

Activity