These are chat archives for arnaud-lb/php-rdkafka

23rd
Oct 2017
Robert Slootjes
@slootjes
Oct 23 2017 07:47
I'm just messing around with kafka now and this extension (thanks a lot for it!) but I run into something I don't understand. When I have a topic with 5 partitions and 2 consumers on the same group I only get 40% (2/5) of the messages and the rest is "lost"
what should I do to get all messages? When using 2 partitions and 2 consumers everything goes well btw.
jstean
@jstean
Oct 23 2017 09:20
@slootjes are you sure they are in the same group? you can use kafka-consumer-groups.sh in your kafka install to inspect
Robert Slootjes
@slootjes
Oct 23 2017 09:20
they are in the same group, but 2 consumers consume 2 partitions, not all 5 of them
I'm using the free Cloudkarafka plan to test with so I don't have access to server side tools right now
Robert Slootjes
@slootjes
Oct 23 2017 12:25
just tried it again and now it works fine...
Robert Slootjes
@slootjes
Oct 23 2017 13:51
I think I know what it is... I stopped 1 of the 2 consumers and there was no repartioning
so the 1 consumer that I have left just gets 2 out of 5 topics and I lose the rest
for some reason no repartioning is taking place
Robert Slootjes
@slootjes
Oct 23 2017 14:57
When killing a process (ctrl+c) somehow kafka doesn't seem to take it out of the pool directly
so when I kill it and rejoin I sometimes end up with 3 out of 5 partitions not being asigned to a consumer...how can I prevent this?