hey @edenhill - do you know how librdkafka is handled in an FPM environment? ie: does it instantiate the librarye for each “worker” (handling up to x thouhsand connections using the same instantiation) or does every process fire up the library? Also, does it handle queuing per library, or per application - ie: if I set queuing to 1000 before sending is that 1000 per application or 1000 stored in the lib before they are sent (or somewhere locally?) trying to optimise performance and resources
I have no idea, I don't use PHP. Since there is a startup and termination cost to each kafka client instance I would recommend reusing them as far as possible.
yeah that’s what I’m hoping to do - in a “normal” FPM set up, it’s common to have low workers to handle a high amount of requests to avoid that issue - I just wondered how it worked with librdkafka :) thanks for responding
different note @edenhill - using the
rdkafka_performance function in the examples directory, how / where do I define the hosts / brokers to send the messages to?
-b <broker1,broker2>
try rdkafka_performance -h for help
perfect, thank you
ahhh I was checking the source - i’ll go for that superb - thanks
Make sure to buckle up before you run it!
:D
(that will inevitably come back and bite me)
lol
hmmm, slow, but that could be our Kafka set up :/
% Sending 500000 messages of size 100 bytes
% 500000 messages produced (50000000 bytes), 0 delivered (offset 0, 0 failed) in 1000ms: 0 msgs/s and 0.00 Mb/s, 0 produce failures, 500000 in queue, no compression
% 500000 messages produced (50000000 bytes), 500000 delivered (offset 0, 0 failed) in 1809ms: 276372 msgs/s and 27.64 Mb/s, 0 produce failures, 0 in queue, no compression
try more messages to let it ramp up
5 M messages
yeah I’ll give it a go - I’m trying to replicate how we’re using it and we’re sending like 1 or 2 messages from each app per instantiation
any thoughts why this is happening:
./rdkafka_performance -P -t test2 -s 1500 -c 500000 -v -l -u -m "_____________Test1:TwoBrokers:500kmsgs:1500bytes" -S 1 -a 1 -b 192.168.50.194,192.168.50.195
% Sending 500000 messages of size 30 bytes
ignoring the size options
not seen this tool before
looks interesting
also, am I right in thinking
-p should set the number of partitions for the topic? or is it a specific partition?
@cpats007 -p sets the partition to produce to, if you leave it out it will use the default partitioner to select partition
I'm not sure why it misses -s, weird
oh, right, -m sets a pattern, ignoring -s.
so the length will be the length of the pattern
no, it must have been some other reason, the -s is working now, so prehaps I had the configs set up wierdly or something