thanks @edenhill - I've already set it to 1ms, but just going to upgrade to version 0.9.4 and see if that helps. Also, the way I was calling poll was with the following:
$this->getProducer()->poll(-1);
which from what I can gather waits until it gets a response. I tried changing to a while loop of $producer->poll(10); etc but didn't see any difference - so are there any implications in using my original methodology of $this->getProducer()->poll(-1); or any reasons not to do it that way etc?
my configs are:
$conf->set('client.id', 'track');
$conf->set('socket.blocking.max.ms', 1);
$conf->set('queue.buffering.max.ms', 1);
$conf->set('batch.num.messages', 1);
Maybe thats the network + broker latency? If you enable stats and check the per-broker average rtt (see https://github.com/edenhill/librdkafka/wiki/Statistics), or enable debug=protocol and look for rtt printouts of the ProduceRequests you will get an idea of that latency
I also suggest you not to use batch.num.messages=1, there is really no point in forcing single message batches, it wil only (potentially) slow things down. The batching is already limited by queue.buffering.max.ms
if that's not implemented, go with the debug=protocol approach
does that output to anywhere in particular?
but this is before doing the changes above
I have no idea how to get statistics in the PHP client lol, and nothing in the error logs from FPM or anything
unless you are running it from some web server thingie, in that case you
will probably need to do something else. Im not a php man so I wouldnt know
I can’t even set a stats_cb as a config value as it throws the following error:
RdKafka\Exception: Property "stats_cb" must be set through dedicated .._set_..() function
Anyone able to explain how the Kafka batch mesdsaging works in an FPM environment? ie: does the librdkafka batch messages or are they batched per application? Let’s say I have 100 fpm processes running each producing messages etc, if they are set to batch will they batch on a library level or per application / worker?
Prod is using docker with AlpineOS - I assume there isn't any alpine packages for your C lib ? :)
I've put this together
# Kafka Installation
RUN apk update && apk add make g++ python autoconf php5-dev && \
# librdkafka
cd /tmp && git clone --depth 1 --branch v0.9.4 https://github.com/edenhill/librdkafka.git && \
cd /tmp/librdkafka && ./configure && make && make install && rm -rf /tmp/librdkafka && \
# php-rdkafka
cd /tmp && git clone --depth 1 --branch 3.0.1 https://github.com/arnaud-lb/php-rdkafka.git && \
cd /tmp/php-rdkafka && phpize && ./configure && make all -j 5 && make install && \
echo "extension=rdkafka.so" > /etc/php5/conf.d/rdkafka.ini && \
rm -rf /tmp/php-rdkafka
If I moved it to a bash script, then indeed I could use -e :) but i'm lazy
hey @edenhill - do you know how librdkafka is handled in an FPM environment? ie: does it instantiate the librarye for each “worker” (handling up to x thouhsand connections using the same instantiation) or does every process fire up the library? Also, does it handle queuing per library, or per application - ie: if I set queuing to 1000 before sending is that 1000 per application or 1000 stored in the lib before they are sent (or somewhere locally?) trying to optimise performance and resources
try rdkafka_performance -h for help
ahhh I was checking the source - i’ll go for that superb - thanks