These are chat archives for arnaud-lb/php-rdkafka

15th
Oct 2018
bbaga
@bbaga
Oct 15 2018 08:50

Hello, I'm playing round with:

rdkafka 3.0.5
librdkafka 0.11.3-1build1
php-fpm 7.2

and the producer seems pretty slow. I get 0.2s - 0.3s response times when producing a single message. The ping between the kafka server and the fpm box is under 1 ms.
I set:

$conf = new \RdKafka\Conf();
$conf->set('socket.blocking.max.ms', 1);
$conf->set('queue.buffering.max.ms', 1);
$conf->set('queue.buffering.max.messages', 10);

that's how I got under 1s response times, but still not ideal.

I tried to look into the process with strace and it seems like it is doing nanosleeps left and right:

[pid  8733] 1539590725.125394 nanosleep({tv_sec=0, tv_nsec=100000000},  <unfinished ...>
[pid  8734] 1539590725.135438 <... nanosleep resumed> 0x7fa3689717b0) = 0
[pid  8734] 1539590725.135625 poll([{fd=10, events=POLLIN}, {fd=7, events=POLLIN}], 2, 0) = 1 ([{fd=10, revents=POLLIN}])
[pid  8734] 1539590725.135914 recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0\0\0r\0\0\0\2", iov_len=8}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 8
[pid  8734] 1539590725.136162 recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0\0\0\1\0\0\0\0\0\r rtt\0\0#\204\377\377\0\26MBGHwnWXRhedsHm5z8l3xQ\0\0\0\0\0\0\0\1\0\0\0\16myAmazingTopic\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0", iov_len=110}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 110
[pid  8734] 1539590725.136428 write(2, "%7|1539590725.136|RECV|rdkafka#producer-7| [thrd:xxxxx:9092/bootstrap]: xxxxx:9092/bootstrap: Received MetadataResponse (v2, 110 bytes, CorrId 2, rtt 101.12ms)\n", 176) = 176
...
[pid  8732] 1539590725.139917 write(2, "%7|1539590725.139|UAS|rdkafka#producer-7| [thrd:main]: 1/1 messages were partitioned in topic myAmazingTopic\n", 109) = 109
[pid  8732] 1539590725.140157 write(2, "%7|1539590725.140|METADATA|rdkafka#producer-7| [thrd:main]: xxxxx:9092/bootstrap: 1/1 requested topic(s) seen in metadata\n", 130) = 130
[pid  8732] 1539590725.140328 write(2, "%7|1539590725.140|CLUSTERID|rdkafka#producer-7| [thrd:main]: xxxxx:9092/bootstrap: ClusterId update \"\" -> \"MBGHwnWXRhedsHm5z8l3xQ\"\n", 139) = 139
[pid  8732] 1539590725.140497 futex(0x559716d7dcd4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1539590726, tv_nsec=22485000}, 0xffffffff <unfinished ...>
[pid  8733] 1539590725.225731 <... nanosleep resumed> 0x7fa3691727b0) = 0

Any idea how to reduce those sleep times? Did somebody manage to make it more performant than this or that is the best I'll get from librdkafka?