These are chat archives for arnaud-lb/php-rdkafka

23rd
Jun 2016
Fabien
@fabienduval
Jun 23 2016 07:13
Hello guys, I am experiencing a problem by using php-kafka from apache2 requests. Some of my api requests produce messages to kafka, the trouble is that, each time, the kafka destructor (PHP_METHOD(RdKafkaKafka, destruct)) takes more than 1.5 seconds, sometimes up to 3 seconds. We found that the problem comes from that loop "while (rd_kafka_outq_len(intern->rk) > 0)" (2/3 of the waiting time) and that call "rd_kafka_destroy" (1/3 of the waiting time).
I tried various parameters for the produced message to be snt immediately so that there is no waiting time in the destructor, but I could not find anyting relevant. Did some of you already experience that kind of situation? If so, how did you resolve it?
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 08:12
Hi Fabien
Fabien
@fabienduval
Jun 23 2016 08:21
Hi Arnaud, yes I tried it (in a script, not in an apache2 request), but the thread termination took the same amount of time.
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 08:21
Could you share the script?
Fabien
@fabienduval
Jun 23 2016 08:29
the C (modified driver to keep track of the time on stderr) : http://pastebin.com/trL1LNaQ
And the result of the execution : http://pastebin.com/HjGFx3AT
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 08:34
What versions of PHP, librdkafka, php-rdkafka, Kafka are you using ?
Fabien
@fabienduval
Jun 23 2016 08:35
librdkafka : confluent-debian-0.9.1
php-rdkafka : I am on master
php : 5.5.9
Kafka 0.9.0.1
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 08:55
Thanks
reducing socket.blocking.max.ms should reduce destruction time
Fabien
@fabienduval
Jun 23 2016 08:56
Wow
indeed
It is quiet drastic
thanks
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 08:57
Welcome :)
Arnaud Le Blanc
@arnaud-lb
Jun 23 2016 09:07
This setting is used as a sleep time in this loop that checks for termination: https://github.com/edenhill/librdkafka/blob/0ff40cc08bc6c3ba35cc5205c91131618a58a0c7/src/rdkafka.c#L975
Fabien
@fabienduval
Jun 23 2016 09:23
I am not sure to fully understand that code, I will dig into it a bit later
thanks