These are chat archives for twitter/summingbird

26th
May 2017
Timur
@ttim
May 26 2017 02:10
Is it true that if durationMillis for Batcher == 1 then BatchId == millisecond ?
Even for negative values?
And if durationMillis == 2, then for batchId == -1 milliseconds will be -1 and -2
P. Oscar Boykin
@johnynek
May 26 2017 02:13
@ttim I don't follow your question
Timur
@ttim
May 26 2017 02:13
@johnynek imagine I use MillisecondBatcher(1), then what is intended value for batchOf(-1) ?
P. Oscar Boykin
@johnynek
May 26 2017 02:14
so, 1) I guess you can check at the repl what it does.
2) we don't intend for anyone to use 1 ms batches.
Timur
@ttim
May 26 2017 02:14
or if I use MillisecondBatcher(2), then what is intended value for batchOf(-1 & -2)
P. Oscar Boykin
@johnynek
May 26 2017 02:14
I don't think hadoop can keep up. ;)
we could just throw if you like.
Timur
@ttim
May 26 2017 02:14
I know what it does, and that's why I'm wondering what is intended =)
P. Oscar Boykin
@johnynek
May 26 2017 02:14
say, less than 1 minute is invalid
what is intended is to divide by a fixed batch width of unix time.
Timur
@ttim
May 26 2017 02:15
I'll say timestamps less than 0 is also highly unlikely =)
P. Oscar Boykin
@johnynek
May 26 2017 02:15
yep.
baring some amazing backfills
Timur
@ttim
May 26 2017 02:15
but that's another thing which failed my tests >_<
P. Oscar Boykin
@johnynek
May 26 2017 02:15
it didn't come up that I notice.
there could be a real bug. There are some well specified bather laws.
I think they all make sense
I wouldn't be quick to change those laws.
but the batchers may have bugs at negative times.
if we can easily fix them, why not.
Timur
@ttim
May 26 2017 02:17
yep, so that's why I asked about intended behaviour in the first place. I guess I fixed them, just wondering did I fix them in a right way =)
P. Oscar Boykin
@johnynek
May 26 2017 02:17
if you didn't change the batcher laws,
I think it's okay
Timur
@ttim
May 26 2017 02:17
sounds good
P. Oscar Boykin
@johnynek
May 26 2017 02:17
:thumbsup:
=> clocking out
Timur
@ttim
May 26 2017 23:23
I have a feeling like current MemoryPlatform isn't useful, and we should implement the one which behaves like Scalding with timestamped events. It would be useful for tests, what do you think @johnynek ?