These are chat archives for paypal/squbs
delayseems to be pretty tricky in
TimeoutFlowat https://github.com/paypal/squbs/blob/master/squbs-ext/src/main/scala/org/squbs/streams/StreamTimeout.scala#L67. The default buffer size is 16 and OverflowStrategy is
dropTail. I think for the logic here, we cannot use any
droptype overflow strategy and should instead use
backpressure. When I set delay buffer to a higher number, batching increases and stages execute in batches. For instance,
partitiongets executed for many elements then feedback gets executed for many elements. This prevents us updating the CB state timely and results in CB not deciding properly. When I decrease the buffer size (e.g., to 2), I see a good CB behavior, but now the throughput decreases significantly.
Atomic. For the examples, I use
Atomicfor now. This is about not getting to the code logic where we update the CB state (with feedback loop or
Atomic) timely. Basically, to increase throughput, we end up getting too many false positives and too many false negatives. There should be a way of managing the concurrency semantics properly in the stream, but, it might be quite difficult. Also we cannot know (and assume) application usages of CB and timeout flow.