These are chat archives for arnaud-lb/php-rdkafka

2nd
Mar 2017
Craig Patrick
@cpats007
Mar 02 2017 14:53

thanks @edenhill - I've already set it to 1ms, but just going to upgrade to version 0.9.4 and see if that helps. Also, the way I was calling poll was with the following:

$this->getProducer()->poll(-1);

which from what I can gather waits until it gets a response. I tried changing to a while loop of $producer->poll(10); etc but didn't see any difference - so are there any implications in using my original methodology of $this->getProducer()->poll(-1); or any reasons not to do it that way etc?

Magnus Edenhill
@edenhill
Mar 02 2017 15:38
produce(); poll(-1); will not necessarily serve the delivery report of the message produced by that specific call to produce(). It may return sooner than that because of other callbacks (stats, error ,.etc)
Craig Patrick
@cpats007
Mar 02 2017 15:39

ah I see, so it’s best to use the loop method?

also, I’ve just upgraded to 0.9.4 - ran a few quick tests and still getting around 30ms for Kafka

my configs are:
$conf->set('client.id', 'track');
$conf->set('socket.blocking.max.ms', 1);
$conf->set('queue.buffering.max.ms', 1);
$conf->set('batch.num.messages', 1);
Magnus Edenhill
@edenhill
Mar 02 2017 15:40
Maybe thats the network + broker latency? If you enable stats and check the per-broker average rtt (see https://github.com/edenhill/librdkafka/wiki/Statistics), or enable debug=protocol and look for rtt printouts of the ProduceRequests you will get an idea of that latency
I also suggest you not to use batch.num.messages=1, there is really no point in forcing single message batches, it wil only (potentially) slow things down. The batching is already limited by queue.buffering.max.ms
Craig Patrick
@cpats007
Mar 02 2017 15:42

okay, thanks for that recommendation i’ll change it

how would I go about enabling stats in PHP?

Magnus Edenhill
@edenhill
Mar 02 2017 15:43
not sure as I havent used that client myself, but look for something resembling, stats_cb, stats, statistics, ..
if that's not implemented, go with the debug=protocol approach
Craig Patrick
@cpats007
Mar 02 2017 15:44

okay, appreciate that - thanks

how do I set the debug? ;)

Magnus Edenhill
@edenhill
Mar 02 2017 15:50
conf..set('debug', 'protocol');
Craig Patrick
@cpats007
Mar 02 2017 15:50
thanks @edenhill - appreciate the help
does that output to anywhere in particular?
Magnus Edenhill
@edenhill
Mar 02 2017 15:59
stderr by default
Craig Patrick
@cpats007
Mar 02 2017 16:02
thanks
Craig Patrick
@cpats007
Mar 02 2017 16:19

I can see a couple of these curently in my logs:

Unable to write data to Kafka: Local: Message timed out

but this is before doing the changes above
Magnus Edenhill
@edenhill
Mar 02 2017 16:46
is that from your delivery report callback?
Craig Patrick
@cpats007
Mar 02 2017 16:46
yeah
I have no idea how to get statistics in the PHP client lol, and nothing in the error logs from FPM or anything
Magnus Edenhill
@edenhill
Mar 02 2017 16:48
debug=protocol is easier
Craig Patrick
@cpats007
Mar 02 2017 16:49
I’ve got $conf->set('debug', 'protocol’); but again, no logs and unable to get the stderr output - unless I’m missing something?
Magnus Edenhill
@edenhill
Mar 02 2017 16:49
should just print to the terminal
unless you are running it from some web server thingie, in that case you
will probably need to do something else. Im not a php man so I wouldnt know
Craig Patrick
@cpats007
Mar 02 2017 16:51
yeah this is a web based application, so is running through php-fpm - I’ve no idea how to do it lol :D

I can’t even set a stats_cb as a config value as it throws the following error:

RdKafka\Exception: Property "stats_cb" must be set through dedicated .._set_..() function