These are chat archives for arnaud-lb/php-rdkafka

9th
Mar 2017
Craig Patrick
@cpats007
Mar 09 2017 11:24
hey @edenhill - do you know how librdkafka is handled in an FPM environment? ie: does it instantiate the librarye for each “worker” (handling up to x thouhsand connections using the same instantiation) or does every process fire up the library? Also, does it handle queuing per library, or per application - ie: if I set queuing to 1000 before sending is that 1000 per application or 1000 stored in the lib before they are sent (or somewhere locally?) trying to optimise performance and resources
Magnus Edenhill
@edenhill
Mar 09 2017 11:28
I have no idea, I don't use PHP. Since there is a startup and termination cost to each kafka client instance I would recommend reusing them as far as possible.
Craig Patrick
@cpats007
Mar 09 2017 11:29
yeah that’s what I’m hoping to do - in a “normal” FPM set up, it’s common to have low workers to handle a high amount of requests to avoid that issue - I just wondered how it worked with librdkafka :) thanks for responding
Craig Patrick
@cpats007
Mar 09 2017 12:14
different note @edenhill - using the rdkafka_performance function in the examples directory, how / where do I define the hosts / brokers to send the messages to?
Magnus Edenhill
@edenhill
Mar 09 2017 12:15
-b <broker1,broker2>
try rdkafka_performance -h for help
Craig Patrick
@cpats007
Mar 09 2017 12:15
perfect, thank you
ahhh I was checking the source - i’ll go for that superb - thanks
Magnus Edenhill
@edenhill
Mar 09 2017 12:15
Make sure to buckle up before you run it!
Craig Patrick
@cpats007
Mar 09 2017 12:15
:D
Magnus Edenhill
@edenhill
Mar 09 2017 12:15
(that will inevitably come back and bite me)
Craig Patrick
@cpats007
Mar 09 2017 12:16
lol
Craig Patrick
@cpats007
Mar 09 2017 12:24
hmmm, slow, but that could be our Kafka set up :/
% Sending 500000 messages of size 100 bytes
% 500000 messages produced (50000000 bytes), 0 delivered (offset 0, 0 failed) in 1000ms: 0 msgs/s and 0.00 Mb/s, 0 produce failures, 500000 in queue, no compression
% 500000 messages produced (50000000 bytes), 500000 delivered (offset 0, 0 failed) in 1809ms: 276372 msgs/s and 27.64 Mb/s, 0 produce failures, 0 in queue, no compression
Magnus Edenhill
@edenhill
Mar 09 2017 12:41
try more messages to let it ramp up
5 M messages
Craig Patrick
@cpats007
Mar 09 2017 12:42
yeah I’ll give it a go - I’m trying to replicate how we’re using it and we’re sending like 1 or 2 messages from each app per instantiation
Craig Patrick
@cpats007
Mar 09 2017 12:47
any thoughts why this is happening:
./rdkafka_performance -P -t test2 -s 1500 -c 500000 -v -l -u -m "_____________Test1:TwoBrokers:500kmsgs:1500bytes" -S 1 -a 1 -b 192.168.50.194,192.168.50.195
% Sending 500000 messages of size 30 bytes
ignoring the size options
Paul Dragoonis
@dragoonis
Mar 09 2017 13:09
not seen this tool before
looks interesting
Craig Patrick
@cpats007
Mar 09 2017 13:18
also, am I right in thinking -p should set the number of partitions for the topic? or is it a specific partition?
Magnus Edenhill
@edenhill
Mar 09 2017 13:34
@cpats007 -p sets the partition to produce to, if you leave it out it will use the default partitioner to select partition
I'm not sure why it misses -s, weird
oh, right, -m sets a pattern, ignoring -s.
so the length will be the length of the pattern
Craig Patrick
@cpats007
Mar 09 2017 13:40
no, it must have been some other reason, the -s is working now, so prehaps I had the configs set up wierdly or something