These are chat archives for linkedin/pinot

17th
Feb 2016
DaiYu
@daifish
Feb 17 2016 08:50
@jfim hi, when pinot run the realtime task, i see only use AvroSchemaToPinotSchema and addable command, and query is ok, and i wonder that how the index created in the memory? thanks for your time :)
DaiYu
@daifish
Feb 17 2016 10:48
btw, we want to use the RocketMQ to replace the kafka in the pinot, can u share some informations about it? :) @jfim
Jean-Fran├žois Im
@jfim
Feb 17 2016 18:01
@daifish Basically we store the memory in a buffer and build an inverted index on the fly; once it reaches a certain threshold we flush it to disk like a normal segment
Jean-Fran├žois Im
@jfim
Feb 17 2016 18:07
As for using a different realtime input source, you can create an adapter for it. We don't use RocketMQ at LinkedIn so it would be hard to test on our end. You need to implement the StreamProvider interface in your own class and specify that as the data source in your realtime table configuration.