These are chat archives for arnaud-lb/php-rdkafka

3rd
Oct 2017
jstean
@jstean
Oct 03 2017 08:22
@Jan051297 alter your topic to have more than 1 partition, if it only has 1 partition http://ronnieroller.com/kafka/cheat-sheet#add-a-partition
jstean
@jstean
Oct 03 2017 09:02
@edenhill would you have any suggestion why some consumers ( in their own unique group ) have "unknown" in their partition describe, looks like they are not getting allocated all the partitions or they never start to consume from the partition for unknown reason?
Magnus Edenhill
@edenhill
Oct 03 2017 09:06
@jstean Where do you see "unknown"?
jstean
@jstean
Oct 03 2017 09:07
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group xyz
Magnus Edenhill
@edenhill
Oct 03 2017 09:07
can you show me?
jstean
@jstean
Oct 03 2017 09:08
@edenhill topic with 10 partitions, some consumers ( in unique groups using high level consumer ) have "unknown" for current offset and lag
@edenhill as a result they dont get all the messages on the topic
Magnus Edenhill
@edenhill
Oct 03 2017 09:09
Depending on Kafka version you might nee to specify --new-consumer to that script
If it still shows up as unknown it means that consumer has not committed offsets for the given partition, or that the committed offsets have been purged by the offset topic retention policy. (1w by default, iirc)
jstean
@jstean
Oct 03 2017 09:10
@edenhill kafka_2.10-0.10.1.1
@edenhill I'm using the default conf with [enable.auto.offset.store] => true and enable.auto.commit] => true
Magnus Edenhill
@edenhill
Oct 03 2017 09:12
If a librdkafka consumer reaches the end of the partition before seeing any real messages it will not commit the offset of the "end offset" of the partition. This differs from the Java client.
jstean
@jstean
Oct 03 2017 09:13
does that mean it just then stops consuming further partitions?
@edenhill its very inconsistent, some times cannot get the "issue" to happen with identical code/properties
Magnus Edenhill
@edenhill
Oct 03 2017 09:18
no, it will keep consuming.
If you can reproduce with debug=cgrp,topic,fetch,broker,protocol then open an issue on librdkafka
jstean
@jstean
Oct 03 2017 09:18
@edenhill ok ty will do that :+1: