These are chat archives for nrinaudo/kantan.csv

25th
Apr 2017
Andrew Roberts
@aroberts
Apr 25 2017 20:35
@nrinaudo RowDecoder appears not to be serializable when crezted via RowDecoder.ordered, due to inheriting GeneratedRowDecoders and possibly other things. Have you encountered this before?
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:42
crezted? what? :)
oh sorry, created
Serializable - as in, you're trying to serialize an instance of RowDecoder?
may I ask why?
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:49
but yes, you're absolutely right, I've just checked and somehow, even though Decoder does extend Serializable, concrete decoders don't appear to. Not sure why
I'm a bit busy just now, would you mind opening an issue about that and I'll take a look as soon as I make some time?
Andrew Roberts
@aroberts
Apr 25 2017 20:51
sure thing
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:51
thanks!
Andrew Roberts
@aroberts
Apr 25 2017 20:51
and it’s an unfortunate side effect of the framework I’m using
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:51
spark?
Andrew Roberts
@aroberts
Apr 25 2017 20:51
flink
but same idea
I’m going to see if I can @transient my way out of it for now
I’m also considering a shapeless-based generic derivation approach
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:52
Alright, understood. I thought I'd addressed that by having Decoder extend Serializable, but apparently that doesn't carry over to descendants
I doubt it'll be any different with shapeless derived instances, it's not actually anything to do with GeneratedRowDecoders
if you create a new instance through RowDecoder.from, it won't be serializable either
Andrew Roberts
@aroberts
Apr 25 2017 20:53
I don’t appear to have this problem with RowDecoder.from, somehow
oh- wait...
right, I had nested my way past this before
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:54
you must have:
scala> RowDecoder.from(ss => DecodeResult.success(1)).isInstanceOf[Serializable]
res6: Boolean = false
Andrew Roberts
@aroberts
Apr 25 2017 20:56
for some reason, flink thinks that RowDecoder.from produces serializable instances
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:56
far be it from me to question flink's sanity, but...
Andrew Roberts
@aroberts
Apr 25 2017 20:57
    val o = QueueEvent.decodeQueueEvent
    assert(o.isInstanceOf[Serializable])
that assert passes
and decodeQueueEvent is derived from RowDecoder.from
/shrug
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:58
decodeQueueEvent is a RowDecoder?
Andrew Roberts
@aroberts
Apr 25 2017 20:58
yes, sorry
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:58
you're doing weird things to my library :)
Andrew Roberts
@aroberts
Apr 25 2017 20:58
it’s true
remember this?
  def discriminatorRowDecoder[C: CellDecoder, A](index: Int)(discriminator: PartialFunction[C, Decode[A]]): RowDecoder[A] =
    RowDecoder.from(input => for {
      data <- input.lift(index).map(CellDecoder[C].decode).getOrElse(DecodeResult.outOfBounds(index))
      discriminated <- discriminator.applyOrElse(data, invalidDiscriminator)(input)
    } yield discriminated)
that’s producing decodeQueueEvent
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 20:59
so... decoders are events, in your system?
Andrew Roberts
@aroberts
Apr 25 2017 21:00
events are being decoded, yes, if that’s what you mean
and each event has its own decoder because the schemas are different (and in many cases even the arity is different, ugh)
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:00
aaah. decodeQueueEvent is used to decode queue events, it is not a decode queue event. Sorry. That makes more sense
Andrew Roberts
@aroberts
Apr 25 2017 21:03
(the missing link:
  implicit val decodeQueueEvent: RowDecoder[QueueEvent] = discriminatorRowDecoder[String, QueueEvent](4) {
    caseABANDON” => RowDecoder.decoder(0,1,2,5,6,7)(Abandon.apply)
    // etc
  }
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:04
it's really, really weird that this would be Serializable. Because RowDecoder.ordered basically calls RowDecoder.from
Andrew Roberts
@aroberts
Apr 25 2017 21:04
right - I completely don’t understand why this is happening like this
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:05
I'll investigate and will let you know - I'm afraid that if you need a workaround right this minute, you're on your own though, I'm on baby watch tonight
Andrew Roberts
@aroberts
Apr 25 2017 21:06
No problem, it’s not urgent
part of a back-burner migration
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:07
this is going to bug me though, what the hell is going on
I have a Scala meetup this week, I'll ask around, there's bound to be someone knowledgeable about this
Andrew Roberts
@aroberts
Apr 25 2017 21:09
cool. I’m doing some experiements to flesh out the ticket
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:09
appreciate it
Andrew Roberts
@aroberts
Apr 25 2017 21:23
nrinaudo/kantan.csv#81, when you get to it
Nicolas Rinaudo
@nrinaudo
Apr 25 2017 21:26
So - mixing in a trait that's not serializable makes everything non-serializable? That might be worth looking into, thanks!