Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Magnus Edenhill
    @edenhill
    looks great. That's not a balanced consumer though, just a standalone consumer since you use assign() rather than subscribe()
    but that's up to you and your use-case
    Paul Dragoonis
    @dragoonis
    I think, based on my use-case you said I should use assign()
    apocello2008
    @apocello2008
    @edenhill Hello! I wrote a message some days ago, can you see pm please?
    ryanminside
    @ryanminside

    PHP 5.9:

    vagrant@vagrant$ php --version
    PHP 5.5.9-1ubuntu4.20 (cli) (built: Oct  3 2016 13:00:37)
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
        with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies

    php-rdkafka 3.0.1 and librdkafka 0.9:

    vagrant@vagrant$ php --ri rdkafka
    
    rdkafka
    
    rdkafka support => enabled
    version => 3.0.1
    build date => Feb  6 2017 14:43:27
    librdkafka version (runtime) => 0.9.2-291-g281ca3
    librdkafka version (build) => 0.9.4.0
    Eliurkis
    @eliurkis_twitter
    People, I am trying to do:
    sudo pecl install channel://pecl.php.net/rdkafka
    But I am getting a bunch of errors like:
    /tmp/pear/install/rdkafka/rdkafka.c:662:19: note: each undeclared identifier is reported only once for each function it appears in
         COPY_CONSTANT(RD_KAFKA_RESP_ERR__ISR_INSUFF);
                       ^
    /usr/include/php/20151012/Zend/zend_constants.h:42:105: note: in definition of macro ‘REGISTER_LONG_CONSTANT’
     #define REGISTER_LONG_CONSTANT(name, lval, flags)  zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
                                                                                                             ^
    /tmp/pear/install/rdkafka/rdkafka.c:662:5: note: in expansion of macro ‘COPY_CONSTANT’
         COPY_CONSTANT(RD_KAFKA_RESP_ERR__ISR_INSUFF);
         ^
    make: *** [rdkafka.lo] Error 1
    ERROR: `make' failed
    Magnus Edenhill
    @edenhill
    @eliurkis_twitter Is your librdkafka version too old perhaps?
    Eliurkis
    @eliurkis_twitter
    @edenhill
    vagrant@homestead:~/php-rdkafka$ sudo apt-get install librdkafka1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    librdkafka1 is already the newest version.
    Look like it's up-to-date
    Eliurkis
    @eliurkis_twitter
    Any idea?
    Magnus Edenhill
    @edenhill
    what version is that?
    Eliurkis
    @eliurkis_twitter
    vagrant@homestead:~/php-rdkafka$ sudo apt-cache policy librdkafka1
    librdkafka1:
      Installed: 0.8.3-1ubuntu2
      Candidate: 0.8.3-1ubuntu2
      Version table:
     *** 0.8.3-1ubuntu2 0
            500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
            100 /var/lib/dpkg/status
    @edenhill
    Magnus Edenhill
    @edenhill
    whoa, that's very old
    Eliurkis
    @eliurkis_twitter
    It's the one that have Ubuntu on the repo
    Magnus Edenhill
    @edenhill
    You want 0.9.1 or later (0.9.3 was just pushed to latest ubuntu)
    it should be safer than adding a newer ubuntu repo
    Eliurkis
    @eliurkis_twitter
    @edenhill Awesome! It's working. Thank you so much!
    ksachdeva11
    @ksachdeva11

    After running
    php55-rdkafka
    command, what do I do next?

    Thanks!

    Im using mac
    Ana-Maria Lazar
    @anamarialazar
    @edenhill I am getting "Uncaught PHP Exception RdKafka\Exception No such configuration property: socket.blocking.max.ms". I am trying to improve the speed in an FPM env as described in arnaud-lb/php-rdkafka#101
    Magnus Edenhill
    @edenhill
    @anamarialazar Looks like an old librdkafka version. What version are you on?
    Ana-Maria Lazar
    @anamarialazar
    librdkafka1:
    Installed: 0.8.3-1ubuntu2
    Magnus Edenhill
    @edenhill
    That's really old. Upgrade to 0.9.2 or 0.9.3
    Ana-Maria Lazar
    @anamarialazar
    Thanks
    Craig Patrick
    @cpats007
    Hi all - anyone know the best way to reduce latency in PHP talking to Kafka? I'm currently running at around 30ms for just the process of writing data to Kafka - seems like a long time to me?
    Magnus Edenhill
    @edenhill
    @cpats007 use librdkafka 0.9.4 and set queue.buffering.max.ms to your liking
    Craig Patrick
    @cpats007

    thanks @edenhill - I've already set it to 1ms, but just going to upgrade to version 0.9.4 and see if that helps. Also, the way I was calling poll was with the following:

    $this->getProducer()->poll(-1);

    which from what I can gather waits until it gets a response. I tried changing to a while loop of $producer->poll(10); etc but didn't see any difference - so are there any implications in using my original methodology of $this->getProducer()->poll(-1); or any reasons not to do it that way etc?

    Magnus Edenhill
    @edenhill
    produce(); poll(-1); will not necessarily serve the delivery report of the message produced by that specific call to produce(). It may return sooner than that because of other callbacks (stats, error ,.etc)
    Craig Patrick
    @cpats007

    ah I see, so it’s best to use the loop method?

    also, I’ve just upgraded to 0.9.4 - ran a few quick tests and still getting around 30ms for Kafka

    my configs are:
    $conf->set('client.id', 'track');
    $conf->set('socket.blocking.max.ms', 1);
    $conf->set('queue.buffering.max.ms', 1);
    $conf->set('batch.num.messages', 1);
    Magnus Edenhill
    @edenhill
    Maybe thats the network + broker latency? If you enable stats and check the per-broker average rtt (see https://github.com/edenhill/librdkafka/wiki/Statistics), or enable debug=protocol and look for rtt printouts of the ProduceRequests you will get an idea of that latency
    I also suggest you not to use batch.num.messages=1, there is really no point in forcing single message batches, it wil only (potentially) slow things down. The batching is already limited by queue.buffering.max.ms
    Craig Patrick
    @cpats007

    okay, thanks for that recommendation i’ll change it

    how would I go about enabling stats in PHP?

    Magnus Edenhill
    @edenhill
    not sure as I havent used that client myself, but look for something resembling, stats_cb, stats, statistics, ..
    if that's not implemented, go with the debug=protocol approach
    Craig Patrick
    @cpats007

    okay, appreciate that - thanks

    how do I set the debug? ;)

    Magnus Edenhill
    @edenhill
    conf..set('debug', 'protocol');
    Craig Patrick
    @cpats007
    thanks @edenhill - appreciate the help
    does that output to anywhere in particular?
    Magnus Edenhill
    @edenhill
    stderr by default
    Craig Patrick
    @cpats007
    thanks
    Craig Patrick
    @cpats007

    I can see a couple of these curently in my logs:

    Unable to write data to Kafka: Local: Message timed out

    but this is before doing the changes above
    Magnus Edenhill
    @edenhill
    is that from your delivery report callback?
    Craig Patrick
    @cpats007
    yeah
    I have no idea how to get statistics in the PHP client lol, and nothing in the error logs from FPM or anything
    Magnus Edenhill
    @edenhill
    debug=protocol is easier