These are chat archives for akkadotnet/AkkaStreams

24th
Nov 2016
Amri Hidayat
@amri
Nov 24 2016 09:16
I'm currently working on ActorPublisher to get a stream of database data. Would like to know your suggestion on the mechanism to retrieve the data from the database ? so that I could perform backpressure.
Bartosz Sypytkowski
@Horusiath
Nov 24 2016 09:24

@amri depending on the database i.e. Redis has support for pub/sub, and MongoDB has tailable cursors, so they can be made push-based (and to support backpressure, they need to expose api to limit upstream). But usually most databases offer only pull model (so you'd need to call a database to retrieve data). I've created an example publisher working with Github HTTP API .

Basically when the consumer request for a data, I'm downloading a page of issues from github and buffer it. If the page is bigger than the request demands, I serve events to the stream from in-memory buffer, and make another HTTP request only when the demand if bigger than number of buffered events. This way I can prefetch some data inside the buffer, so I don't need make a HTTP request every time consumer asks for more, but I also don't need to download a whole result set into memory.

Amri Hidayat
@amri
Nov 24 2016 09:29
ahhh I'm clear now. Thanks
Amri Hidayat
@amri
Nov 24 2016 13:13
@Horusiath how do you make sure that your next call to the HTTP API are not duplicates of previous calls ?
Bartosz Sypytkowski
@Horusiath
Nov 24 2016 14:32
@amri I keep page size constant and keep track of the last page requested and the last page actually fetched