These are chat archives for nrinaudo/kantan.csv

25th
Apr 2017
Andrew Roberts
@aroberts
Apr 25 2017 20:35 UTC
@nrinaudo RowDecoder appears not to be serializable when crezted via RowDecoder.ordered, due to inheriting GeneratedRowDecoders and possibly other things. Have you encountered this before?
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:42 UTC
crezted? what? :)
oh sorry, created
Serializable - as in, you're trying to serialize an instance of RowDecoder?
may I ask why?
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:49 UTC
but yes, you're absolutely right, I've just checked and somehow, even though Decoder does extend Serializable, concrete decoders don't appear to. Not sure why
I'm a bit busy just now, would you mind opening an issue about that and I'll take a look as soon as I make some time?
Andrew Roberts
@aroberts
Apr 25 2017 20:51 UTC
sure thing
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:51 UTC
thanks!
Andrew Roberts
@aroberts
Apr 25 2017 20:51 UTC
and it’s an unfortunate side effect of the framework I’m using
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:51 UTC
spark?
Andrew Roberts
@aroberts
Apr 25 2017 20:51 UTC
flink
but same idea
I’m going to see if I can @transient my way out of it for now
I’m also considering a shapeless-based generic derivation approach
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:52 UTC
Alright, understood. I thought I'd addressed that by having Decoder extend Serializable, but apparently that doesn't carry over to descendants
I doubt it'll be any different with shapeless derived instances, it's not actually anything to do with GeneratedRowDecoders
if you create a new instance through RowDecoder.from, it won't be serializable either
Andrew Roberts
@aroberts
Apr 25 2017 20:53 UTC
I don’t appear to have this problem with RowDecoder.from, somehow
oh- wait...
right, I had nested my way past this before
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:54 UTC
you must have:
scala> RowDecoder.from(ss => DecodeResult.success(1)).isInstanceOf[Serializable]
res6: Boolean = false
Andrew Roberts
@aroberts
Apr 25 2017 20:56 UTC
for some reason, flink thinks that RowDecoder.from produces serializable instances
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:56 UTC
far be it from me to question flink's sanity, but...
Andrew Roberts
@aroberts
Apr 25 2017 20:57 UTC
    val o = QueueEvent.decodeQueueEvent
    assert(o.isInstanceOf[Serializable])
that assert passes
and decodeQueueEvent is derived from RowDecoder.from
/shrug
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:58 UTC
decodeQueueEvent is a RowDecoder?
Andrew Roberts
@aroberts
Apr 25 2017 20:58 UTC
yes, sorry
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:58 UTC
you're doing weird things to my library :)
Andrew Roberts
@aroberts
Apr 25 2017 20:58 UTC
it’s true
remember this?
  def discriminatorRowDecoder[C: CellDecoder, A](index: Int)(discriminator: PartialFunction[C, Decode[A]]): RowDecoder[A] =
    RowDecoder.from(input => for {
      data <- input.lift(index).map(CellDecoder[C].decode).getOrElse(DecodeResult.outOfBounds(index))
      discriminated <- discriminator.applyOrElse(data, invalidDiscriminator)(input)
    } yield discriminated)
that’s producing decodeQueueEvent
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:59 UTC
so... decoders are events, in your system?
Andrew Roberts
@aroberts
Apr 25 2017 21:00 UTC
events are being decoded, yes, if that’s what you mean
and each event has its own decoder because the schemas are different (and in many cases even the arity is different, ugh)
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:00 UTC
aaah. decodeQueueEvent is used to decode queue events, it is not a decode queue event. Sorry. That makes more sense
Andrew Roberts
@aroberts
Apr 25 2017 21:03 UTC
(the missing link:
  implicit val decodeQueueEvent: RowDecoder[QueueEvent] = discriminatorRowDecoder[String, QueueEvent](4) {
    caseABANDON” => RowDecoder.decoder(0,1,2,5,6,7)(Abandon.apply)
    // etc
  }
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:04 UTC
it's really, really weird that this would be Serializable. Because RowDecoder.ordered basically calls RowDecoder.from
Andrew Roberts
@aroberts
Apr 25 2017 21:04 UTC
right - I completely don’t understand why this is happening like this
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:05 UTC
I'll investigate and will let you know - I'm afraid that if you need a workaround right this minute, you're on your own though, I'm on baby watch tonight
Andrew Roberts
@aroberts
Apr 25 2017 21:06 UTC
No problem, it’s not urgent
part of a back-burner migration
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:07 UTC
this is going to bug me though, what the hell is going on
I have a Scala meetup this week, I'll ask around, there's bound to be someone knowledgeable about this
Andrew Roberts
@aroberts
Apr 25 2017 21:09 UTC
cool. I’m doing some experiements to flesh out the ticket
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:09 UTC
appreciate it
Andrew Roberts
@aroberts
Apr 25 2017 21:23 UTC
nrinaudo/kantan.csv#81, when you get to it
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:26 UTC
So - mixing in a trait that's not serializable makes everything non-serializable? That might be worth looking into, thanks!