These are chat archives for arnaud-lb/php-rdkafka

27th
Mar 2017
Mangoer
@mangoer-ys
Mar 27 2017 02:51
when set 'offset.store.method' = 'broker' , where is the true place of broker on
borker
who know
Magnus Edenhill
@edenhill
Mar 27 2017 06:04
@mangoer-ys Offsets are written to the __consumer_offsets topic (by the broker)
Mangoer
@mangoer-ys
Mar 27 2017 06:45
@edenhill But I have a doubt that the last modification time of __consumer_offsets in log-file is not modified as a consumer commits offset
and the size of __consumer_offsets file is 0.....
Magnus Edenhill
@edenhill
Mar 27 2017 06:53
@mangoer-ys that's a replicated topic with 50 (by default) partitions
Mangoer
@mangoer-ys
Mar 27 2017 07:02
I'm sorry. I mean the size of log file in __consumer_offsets-0 ~ 49 is 0
Magnus Edenhill
@edenhill
Mar 27 2017 07:07
okay. are you using manual or auto commits?
and have you configured a group.id? (required for broker based commits)
Mangoer
@mangoer-ys
Mar 27 2017 07:21
Oh I know why. It's my error. I found a mistake that I only see the last modification time of consumer_offsets directory is not modified , but only one log file in 50 consumer_offset directory has been modified in fact.
Thank you!
Mangoer
@mangoer-ys
Mar 27 2017 11:27
I found that using kafkaconsumer::commit it sends offset to broker every time as a consumer consume a message. So the log file is very big. How slove the issue?
Magnus Edenhill
@edenhill
Mar 27 2017 11:43
@mangoer-ys Use auto commits instead
Mangoer
@mangoer-ys
Mar 27 2017 12:07
:( must set manual commit in my business
Magnus Edenhill
@edenhill
Mar 27 2017 12:20
@mangoer-ys okay, manual commits are fine, albeit a bit slow, so you should typically avoid doing it per message, or at least do them asynchronously.
Mangoer
@mangoer-ys
Mar 27 2017 12:28
yes, I use commitasync. I expect to batching commit offsets.
Joe Green
@joegreen88
Mar 27 2017 16:03
I am using high level consumer for some period of time, and then I call unsubscribe(), but my php script doesn't terminate
Magnus Edenhill
@edenhill
Mar 27 2017 16:05
make sure to use librdkafka v0.9.4, it fixes a bunch of termination hang corner cases
Neil Young
@nyoung
Mar 27 2017 20:14
@edenhill do you have a timeline for when you are going to cut a tag off the latest master?
Magnus Edenhill
@edenhill
Mar 27 2017 20:16
@nyoung there will probably be a maintenance release this week, maybe not off master though but based on v0.9.5 + cherry-picks
Neil Young
@nyoung
Mar 27 2017 20:22
sounds good