These are chat archives for paypal/squbs

25th
Dec 2016
Dor Ben Dov
@doribd
Dec 25 2016 09:03
@akara hi, took your advice and joined here.
Anil Gursel
@anilgursel
Dec 25 2016 19:54
@akara delay seems to be pretty tricky in TimeoutFlow at https://github.com/paypal/squbs/blob/master/squbs-ext/src/main/scala/org/squbs/streams/StreamTimeout.scala#L67. The default buffer size is 16 and OverflowStrategy is dropTail. I think for the logic here, we cannot use any drop type overflow strategy and should instead use backpressure. When I set delay buffer to a higher number, batching increases and stages execute in batches. For instance, partition gets executed for many elements then feedback gets executed for many elements. This prevents us updating the CB state timely and results in CB not deciding properly. When I decrease the buffer size (e.g., to 2), I see a good CB behavior, but now the throughput decreases significantly.
Anil Gursel
@anilgursel
Dec 25 2016 20:09
Please note, this is not about using feedback loop, or Agent or Atomic. For the examples, I use Atomic for now. This is about not getting to the code logic where we update the CB state (with feedback loop or Agent or Atomic) timely. Basically, to increase throughput, we end up getting too many false positives and too many false negatives. There should be a way of managing the concurrency semantics properly in the stream, but, it might be quite difficult. Also we cannot know (and assume) application usages of CB and timeout flow.
I wonder if it is time to start discussion at akka/akka#15307.