These are chat archives for twitter/summingbird

21st
Aug 2017
P. Oscar Boykin
@johnynek
Aug 21 2017 21:23
@piyushnarang @ttim twitter/summingbird#745
can I get a green light there?
Piyush Narang
@piyushnarang
Aug 21 2017 21:29
@johnynek Timur is out on vacation this week so I think he might not take a look soon. I can try and take a look. I’m not super familiar with SB code so it would be nice to loop in Pankaj too. He’d probably have more detailed feedback
P. Oscar Boykin
@johnynek
Aug 21 2017 21:30
okay. This is a bugfix, with more test coverage of code that is not actually exercised storm or scalding summingbird, so the risk is very minor.
(but no one knows this code but me.... so....)
Piyush Narang
@piyushnarang
Aug 21 2017 21:31
ok cool, I’ll take a look today. Hadn’t checked out the review so wasn’t sure what it touched
haha self +1 ;-)
P. Oscar Boykin
@johnynek
Aug 21 2017 21:31
ok. Thanks.
:)
Pankaj Gupta
@pankajroark
Aug 21 2017 21:57
Does anyone know how to use OrderedSerialization with Summingbird? I believe adding RequiredBinaryComparator trait to BatchedStore should do it.
P. Oscar Boykin
@johnynek
Aug 21 2017 21:58
I don't think that will. That trait is for job
you can set the configuration flag.
Pankaj Gupta
@pankajroark
Aug 21 2017 21:59
I see, is adding that flag all that’s needed?
P. Oscar Boykin
@johnynek
Aug 21 2017 21:59
that makes the scalding job fail if any Ordering is not an OrderedSerialization.
the user still has to import the scroogeOrdSer method:
Pankaj Gupta
@pankajroark
Aug 21 2017 22:00
I see, so user needs to import scroogeOrdSer methods directly in the BatchedStore code.
That’s where Ordering seems to be used in Summingbird scalding platform
and in service
Pankaj Gupta
@pankajroark
Aug 21 2017 22:01
gotcha thanks
P. Oscar Boykin
@johnynek
Aug 21 2017 22:01
I think the BatchedStore is constructed with VersionedStore usually, that gets the Ordering[K] from where it is instantiated
Pankaj Gupta
@pankajroark
Aug 21 2017 22:01
I see
P. Oscar Boykin
@johnynek
Aug 21 2017 22:01
so, where they create the stores, that's where they need to use the macro
Pankaj Gupta
@pankajroark
Aug 21 2017 22:01
makes sense, thanks a lot
P. Oscar Boykin
@johnynek
Aug 21 2017 22:02
good luck
Pankaj Gupta
@pankajroark
Aug 21 2017 22:02
thanks, OrderedSerialization is really cool, trying to sell it to summingbird users :)
P. Oscar Boykin
@johnynek
Aug 21 2017 22:03
yeah, it makes a pretty huge difference
we built a library for feature engineering on top of summingbird that requires it
Pankaj Gupta
@pankajroark
Aug 21 2017 22:03
I see
P. Oscar Boykin
@johnynek
Aug 21 2017 22:03
that way by the time we pass to summingbird we only have OrderedSerialization
Pankaj Gupta
@pankajroark
Aug 21 2017 22:04
I see, makes sense