arnaud-lb/php-rdkafka

Thin PHP binding for librdkafka - Kafka client

Eliurkis

@eliurkis_twitter

Look like it's up-to-date

Eliurkis

@eliurkis_twitter

Any idea?

Magnus Edenhill

@edenhill

what version is that?

Eliurkis

@eliurkis_twitter

vagrant@homestead:~/php-rdkafka$ sudo apt-cache policy librdkafka1
librdkafka1:
  Installed: 0.8.3-1ubuntu2
  Candidate: 0.8.3-1ubuntu2
  Version table:
 *** 0.8.3-1ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status

@edenhill

Magnus Edenhill

@edenhill

whoa, that's very old

Eliurkis

@eliurkis_twitter

It's the one that have Ubuntu on the repo

Magnus Edenhill

@edenhill

You want 0.9.1 or later (0.9.3 was just pushed to latest ubuntu)

try to add this repository: http://docs.confluent.io/3.1.2/installation.html#installation-apt

it should be safer than adding a newer ubuntu repo

Eliurkis

@eliurkis_twitter

@edenhill Awesome! It's working. Thank you so much!

ksachdeva11

@ksachdeva11

After running
php55-rdkafka
command, what do I do next?

Thanks!

Im using mac

Ana-Maria Lazar

@anamarialazar

@edenhill I am getting "Uncaught PHP Exception RdKafka\Exception No such configuration property: socket.blocking.max.ms". I am trying to improve the speed in an FPM env as described in arnaud-lb/php-rdkafka#101

Magnus Edenhill

@edenhill

@anamarialazar Looks like an old librdkafka version. What version are you on?

Ana-Maria Lazar

@anamarialazar

librdkafka1:
Installed: 0.8.3-1ubuntu2

Magnus Edenhill

@edenhill

That's really old. Upgrade to 0.9.2 or 0.9.3

Ana-Maria Lazar

@anamarialazar

Thanks

Craig Patrick

@cpats007

Hi all - anyone know the best way to reduce latency in PHP talking to Kafka? I'm currently running at around 30ms for just the process of writing data to Kafka - seems like a long time to me?

Magnus Edenhill

@edenhill

@cpats007 use librdkafka 0.9.4 and set queue.buffering.max.ms to your liking

Craig Patrick

@cpats007

thanks @edenhill - I've already set it to 1ms, but just going to upgrade to version 0.9.4 and see if that helps. Also, the way I was calling poll was with the following:

$this->getProducer()->poll(-1);

which from what I can gather waits until it gets a response. I tried changing to a while loop of $producer->poll(10); etc but didn't see any difference - so are there any implications in using my original methodology of $this->getProducer()->poll(-1); or any reasons not to do it that way etc?

Magnus Edenhill

@edenhill

produce(); poll(-1); will not necessarily serve the delivery report of the message produced by that specific call to produce(). It may return sooner than that because of other callbacks (stats, error ,.etc)

Craig Patrick

@cpats007

ah I see, so it’s best to use the loop method?

also, I’ve just upgraded to 0.9.4 - ran a few quick tests and still getting around 30ms for Kafka

my configs are:

$conf->set('client.id', 'track');
$conf->set('socket.blocking.max.ms', 1);
$conf->set('queue.buffering.max.ms', 1);
$conf->set('batch.num.messages', 1);

Magnus Edenhill

@edenhill

Maybe thats the network + broker latency? If you enable stats and check the per-broker average rtt (see https://github.com/edenhill/librdkafka/wiki/Statistics), or enable debug=protocol and look for rtt printouts of the ProduceRequests you will get an idea of that latency

I also suggest you not to use batch.num.messages=1, there is really no point in forcing single message batches, it wil only (potentially) slow things down. The batching is already limited by queue.buffering.max.ms

Craig Patrick

@cpats007

okay, thanks for that recommendation i’ll change it

how would I go about enabling stats in PHP?

Magnus Edenhill

@edenhill

not sure as I havent used that client myself, but look for something resembling, stats_cb, stats, statistics, ..

if that's not implemented, go with the debug=protocol approach

Craig Patrick

@cpats007

okay, appreciate that - thanks

how do I set the debug? ;)

Magnus Edenhill

@edenhill

conf..set('debug', 'protocol');

Craig Patrick

@cpats007

thanks @edenhill - appreciate the help

does that output to anywhere in particular?

Magnus Edenhill

@edenhill

stderr by default

Craig Patrick

@cpats007

thanks

Craig Patrick

@cpats007

I can see a couple of these curently in my logs:

Unable to write data to Kafka: Local: Message timed out

but this is before doing the changes above

Magnus Edenhill

@edenhill

is that from your delivery report callback?

Craig Patrick

@cpats007

yeah

I have no idea how to get statistics in the PHP client lol, and nothing in the error logs from FPM or anything

Magnus Edenhill

@edenhill

debug=protocol is easier

Craig Patrick

@cpats007

I’ve got $conf->set('debug', 'protocol’); but again, no logs and unable to get the stderr output - unless I’m missing something?

Magnus Edenhill

@edenhill

should just print to the terminal

unless you are running it from some web server thingie, in that case you

will probably need to do something else. Im not a php man so I wouldnt know

Craig Patrick

@cpats007

yeah this is a web based application, so is running through php-fpm - I’ve no idea how to do it lol :D

I can’t even set a stats_cb as a config value as it throws the following error:

RdKafka\Exception: Property "stats_cb" must be set through dedicated .._set_..() function

Craig Patrick

@cpats007

Anyone able to explain how the Kafka batch mesdsaging works in an FPM environment? ie: does the librdkafka batch messages or are they batched per application? Let’s say I have 100 fpm processes running each producing messages etc, if they are set to batch will they batch on a library level or per application / worker?

Craig Patrick

@cpats007

also, does it create an asynchronous producer by default?

Paul Dragoonis

@dragoonis

Hey @edenhill that's me getting closer to the delivery of this big data migration project :)

Where communities thrive

People

Repo info

Activity