Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Magnus Edenhill
    @edenhill
    will probably need to do something else. Im not a php man so I wouldnt know
    Craig Patrick
    @cpats007
    yeah this is a web based application, so is running through php-fpm - I’ve no idea how to do it lol :D

    I can’t even set a stats_cb as a config value as it throws the following error:

    RdKafka\Exception: Property "stats_cb" must be set through dedicated .._set_..() function

    Craig Patrick
    @cpats007
    Anyone able to explain how the Kafka batch mesdsaging works in an FPM environment? ie: does the librdkafka batch messages or are they batched per application? Let’s say I have 100 fpm processes running each producing messages etc, if they are set to batch will they batch on a library level or per application / worker?
    Craig Patrick
    @cpats007
    also, does it create an asynchronous producer by default?
    Paul Dragoonis
    @dragoonis
    Hey @edenhill that's me getting closer to the delivery of this big data migration project :)
    Prod is using docker with AlpineOS - I assume there isn't any alpine packages for your C lib ? :)
    Paul Dragoonis
    @dragoonis
    I've put this together
    # Kafka Installation
    RUN apk update && apk add make g++ python autoconf php5-dev && \
        # librdkafka
        cd /tmp && git clone --depth 1 --branch v0.9.4 https://github.com/edenhill/librdkafka.git && \
        cd /tmp/librdkafka && ./configure && make && make install && rm -rf /tmp/librdkafka && \
    
        # php-rdkafka
        cd /tmp && git clone --depth 1 --branch 3.0.1 https://github.com/arnaud-lb/php-rdkafka.git && \
        cd /tmp/php-rdkafka && phpize && ./configure && make all -j 5 && make install && \
        echo "extension=rdkafka.so" > /etc/php5/conf.d/rdkafka.ini && \
        rm -rf /tmp/php-rdkafka
    Magnus Edenhill
    @edenhill
    you can probably use 'set -e' at the start of that thing to avoid all the &&
    Paul Dragoonis
    @dragoonis
    it's a Dockerfile, not a bash script
    If I moved it to a bash script, then indeed I could use -e :) but i'm lazy
    Craig Patrick
    @cpats007
    hey @edenhill - do you know how librdkafka is handled in an FPM environment? ie: does it instantiate the librarye for each “worker” (handling up to x thouhsand connections using the same instantiation) or does every process fire up the library? Also, does it handle queuing per library, or per application - ie: if I set queuing to 1000 before sending is that 1000 per application or 1000 stored in the lib before they are sent (or somewhere locally?) trying to optimise performance and resources
    Magnus Edenhill
    @edenhill
    I have no idea, I don't use PHP. Since there is a startup and termination cost to each kafka client instance I would recommend reusing them as far as possible.
    Craig Patrick
    @cpats007
    yeah that’s what I’m hoping to do - in a “normal” FPM set up, it’s common to have low workers to handle a high amount of requests to avoid that issue - I just wondered how it worked with librdkafka :) thanks for responding
    Craig Patrick
    @cpats007
    different note @edenhill - using the rdkafka_performance function in the examples directory, how / where do I define the hosts / brokers to send the messages to?
    Magnus Edenhill
    @edenhill
    -b <broker1,broker2>
    try rdkafka_performance -h for help
    Craig Patrick
    @cpats007
    perfect, thank you
    ahhh I was checking the source - i’ll go for that superb - thanks
    Magnus Edenhill
    @edenhill
    Make sure to buckle up before you run it!
    Craig Patrick
    @cpats007
    :D
    Magnus Edenhill
    @edenhill
    (that will inevitably come back and bite me)
    Craig Patrick
    @cpats007
    lol
    Craig Patrick
    @cpats007
    hmmm, slow, but that could be our Kafka set up :/
    % Sending 500000 messages of size 100 bytes
    % 500000 messages produced (50000000 bytes), 0 delivered (offset 0, 0 failed) in 1000ms: 0 msgs/s and 0.00 Mb/s, 0 produce failures, 500000 in queue, no compression
    % 500000 messages produced (50000000 bytes), 500000 delivered (offset 0, 0 failed) in 1809ms: 276372 msgs/s and 27.64 Mb/s, 0 produce failures, 0 in queue, no compression
    Magnus Edenhill
    @edenhill
    try more messages to let it ramp up
    5 M messages
    Craig Patrick
    @cpats007
    yeah I’ll give it a go - I’m trying to replicate how we’re using it and we’re sending like 1 or 2 messages from each app per instantiation
    Craig Patrick
    @cpats007
    any thoughts why this is happening:
    ./rdkafka_performance -P -t test2 -s 1500 -c 500000 -v -l -u -m "_____________Test1:TwoBrokers:500kmsgs:1500bytes" -S 1 -a 1 -b 192.168.50.194,192.168.50.195
    % Sending 500000 messages of size 30 bytes
    ignoring the size options
    Paul Dragoonis
    @dragoonis
    not seen this tool before
    looks interesting
    Craig Patrick
    @cpats007
    also, am I right in thinking -p should set the number of partitions for the topic? or is it a specific partition?
    Magnus Edenhill
    @edenhill
    @cpats007 -p sets the partition to produce to, if you leave it out it will use the default partitioner to select partition
    I'm not sure why it misses -s, weird
    oh, right, -m sets a pattern, ignoring -s.
    so the length will be the length of the pattern
    Craig Patrick
    @cpats007
    no, it must have been some other reason, the -s is working now, so prehaps I had the configs set up wierdly or something
    Neil Young
    @nyoung
    heya folks - trying to use the php-rdkafka tool and it seems to be working well for our use case, except I can't figure out how to kill a publish or setup a publish timeout if there is some kind of connection error on all brokers
    it just seems to sit and spin for a long time for me - is there some way to force it to fail from PHP?
    Craig Patrick
    @cpats007
    what is the issue you are having, so if the producer can’t produce, you want it to fail in PHP?
    Neil Young
    @nyoung
    yeah, i don't want the php request to sit and wait forever - i need it to drop the message on the floor and complete
    Craig Patrick
    @cpats007
    does this not work:
    $topicConfig = new TopicConf();
    $topicConfig->set('message.timeout.ms', 1000ms);
    Neil Young
    @nyoung
    trying it
    thought i had been through all the settings - that one seems obvious
    (derp)
    Craig Patrick
    @cpats007
    obviously you’ll need to change it for your code, bu then this:
    /** @var ProducerTopic $kafkaTopic */
    $kafkaTopic = $this->getProducer()->newTopic($topicName, $topicConfig);
    Neil Young
    @nyoung
    yeah
    Craig Patrick
    @cpats007
    :)