@nrinaudo
RowDecoder appears not to be serializable when crezted via RowDecoder.ordered, due to inheriting GeneratedRowDecoders and possibly other things. Have you encountered this before?
crezted? what? :)
oh sorry, created
Serializable - as in, you're trying to serialize an instance of
RowDecoder?
may I ask why?
but yes, you're absolutely right, I've just checked and somehow, even though
Decoder does extend Serializable, concrete decoders don't appear to. Not sure why
I'm a bit busy just now, would you mind opening an issue about that and I'll take a look as soon as I make some time?
sure thing
thanks!
and it’s an unfortunate side effect of the framework I’m using
spark?
flink
but same idea
I’m going to see if I can @transient my way out of it for now
I’m also considering a shapeless-based generic derivation approach
Alright, understood. I thought I'd addressed that by having Decoder extend Serializable, but apparently that doesn't carry over to descendants
I doubt it'll be any different with shapeless derived instances, it's not actually anything to do with
GeneratedRowDecoders
if you create a new instance through
RowDecoder.from, it won't be serializable either
I don’t appear to have this problem with
RowDecoder.from, somehow
oh- wait...
right, I had nested my way past this before
you must have:
scala> RowDecoder.from(ss => DecodeResult.success(1)).isInstanceOf[Serializable]
res6: Boolean = false
for some reason, flink thinks that
RowDecoder.from produces serializable instances
far be it from me to question flink's sanity, but...
val o = QueueEvent.decodeQueueEvent
assert(o.isInstanceOf[Serializable])
that assert passes
and
decodeQueueEvent is derived from RowDecoder.from
/shrug
decodeQueueEvent is a RowDecoder?
yes, sorry
you're doing weird things to my library :)
it’s true
remember this?
def discriminatorRowDecoder[C: CellDecoder, A](index: Int)(discriminator: PartialFunction[C, Decode[A]]): RowDecoder[A] =
RowDecoder.from(input => for {
data <- input.lift(index).map(CellDecoder[C].decode).getOrElse(DecodeResult.outOfBounds(index))
discriminated <- discriminator.applyOrElse(data, invalidDiscriminator)(input)
} yield discriminated)
that’s producing
decodeQueueEvent
so... decoders are events, in your system?
events are being decoded, yes, if that’s what you mean
and each event has its own decoder because the schemas are different (and in many cases even the arity is different, ugh)
aaah.
decodeQueueEvent is used to decode queue events, it is not a decode queue event. Sorry. That makes more sense
(the missing link:
implicit val decodeQueueEvent: RowDecoder[QueueEvent] = discriminatorRowDecoder[String, QueueEvent](4) {
case “ABANDON” => RowDecoder.decoder(0,1,2,5,6,7)(Abandon.apply)
// etc
}
it's really, really weird that this would be
Serializable. Because RowDecoder.ordered basically calls RowDecoder.from
right - I completely don’t understand why this is happening like this
I'll investigate and will let you know - I'm afraid that if you need a workaround right this minute, you're on your own though, I'm on baby watch tonight
No problem, it’s not urgent
part of a back-burner migration
this is going to bug me though, what the hell is going on
I have a Scala meetup this week, I'll ask around, there's bound to be someone knowledgeable about this
cool. I’m doing some experiements to flesh out the ticket
appreciate it
nrinaudo/kantan.csv#81, when you get to it
So - mixing in a trait that's not serializable makes everything non-serializable? That might be worth looking into, thanks!