Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Paul Dragoonis
    @dragoonis
    Do you want me to type bt to get the gdb backtrace? (it's not me running this, it's another team member)
    Sure! Will do
    Magnus Edenhill
    @edenhill
    yah
    Paul Dragoonis
    @dragoonis
    @edenhill 0.92 was segfaulting, we're pulling down latest librdkafka version (runtime) => 0.9.2-257-g65c8bc and it's working OK
    So you've patched it. It may be this. edenhill/librdkafka#850 - I'm getting a bt so we know what issue we ran into though. To close this off :)
    Magnus Edenhill
    @edenhill
    :+1:
    Paul Dragoonis
    @dragoonis
    @edenhill can you tell me when the next stable release is planned?
    Magnus Edenhill
    @edenhill
    0.9.3 is a maint release which is due in about 1-2 weeks
    then there'll be a new feature release in a month or so
    Paul Dragoonis
    @dragoonis
    i have bt
    Magnus Edenhill
    @edenhill
    it'd be great if you want to test the 0.9.3.x branch to make sure the above crash is fixed
    Magnus Edenhill
    @edenhill
    yep
    thanks for the backtrace, it didnt say much though but that's fine, you probably identified the issue anyway.
    Paul Dragoonis
    @dragoonis
    maybe :D We're a high level consumer now (thanks to your help ;))
            // Global config settings - see: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md#global-configuration-properties
            $conf = new ConsumerConf();
            $conf->set("enable.auto.commit", "false");
            $conf->set("socket.blocking.max.ms", 1);
            $conf->set("fetch.wait.max.ms", 10);
            $conf->set('metadata.broker.list', implode(',', $brokers));
            $conf->set('group.id', $this->generateGroupId($topicName, $partition));
    
            $conf->setErrorCb(function ($kafka, $err, $reason) {
                printf("Kafka error: %s (reason: %s)\n", rd_kafka_err2str($err), $reason);
            });
    
            $topicConf = new TopicConf();
            $topicConf->set('auto.offset.reset', 'smallest');
    
            $conf->setDefaultTopicConf($topicConf);
    
            $consumer = new HighLevelConsumer($conf);
            $consumer->assign([
                new TopicPartition($topicName, $partition, RD_KAFKA_OFFSET_STORED)
            ]);
    Magnus Edenhill
    @edenhill
    :100:
    Paul Dragoonis
    @dragoonis
    How's it looking, boss?
    Magnus Edenhill
    @edenhill
    looks great. That's not a balanced consumer though, just a standalone consumer since you use assign() rather than subscribe()
    but that's up to you and your use-case
    Paul Dragoonis
    @dragoonis
    I think, based on my use-case you said I should use assign()
    apocello2008
    @apocello2008
    @edenhill Hello! I wrote a message some days ago, can you see pm please?
    ryanminside
    @ryanminside

    PHP 5.9:

    vagrant@vagrant$ php --version
    PHP 5.5.9-1ubuntu4.20 (cli) (built: Oct  3 2016 13:00:37)
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
        with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies

    php-rdkafka 3.0.1 and librdkafka 0.9:

    vagrant@vagrant$ php --ri rdkafka
    
    rdkafka
    
    rdkafka support => enabled
    version => 3.0.1
    build date => Feb  6 2017 14:43:27
    librdkafka version (runtime) => 0.9.2-291-g281ca3
    librdkafka version (build) => 0.9.4.0
    Eliurkis
    @eliurkis_twitter
    People, I am trying to do:
    sudo pecl install channel://pecl.php.net/rdkafka
    But I am getting a bunch of errors like:
    /tmp/pear/install/rdkafka/rdkafka.c:662:19: note: each undeclared identifier is reported only once for each function it appears in
         COPY_CONSTANT(RD_KAFKA_RESP_ERR__ISR_INSUFF);
                       ^
    /usr/include/php/20151012/Zend/zend_constants.h:42:105: note: in definition of macro ‘REGISTER_LONG_CONSTANT’
     #define REGISTER_LONG_CONSTANT(name, lval, flags)  zend_register_long_constant((name), sizeof(name)-1, (lval), (flags), module_number)
                                                                                                             ^
    /tmp/pear/install/rdkafka/rdkafka.c:662:5: note: in expansion of macro ‘COPY_CONSTANT’
         COPY_CONSTANT(RD_KAFKA_RESP_ERR__ISR_INSUFF);
         ^
    make: *** [rdkafka.lo] Error 1
    ERROR: `make' failed
    Magnus Edenhill
    @edenhill
    @eliurkis_twitter Is your librdkafka version too old perhaps?
    Eliurkis
    @eliurkis_twitter
    @edenhill
    vagrant@homestead:~/php-rdkafka$ sudo apt-get install librdkafka1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    librdkafka1 is already the newest version.
    Look like it's up-to-date
    Eliurkis
    @eliurkis_twitter
    Any idea?
    Magnus Edenhill
    @edenhill
    what version is that?
    Eliurkis
    @eliurkis_twitter
    vagrant@homestead:~/php-rdkafka$ sudo apt-cache policy librdkafka1
    librdkafka1:
      Installed: 0.8.3-1ubuntu2
      Candidate: 0.8.3-1ubuntu2
      Version table:
     *** 0.8.3-1ubuntu2 0
            500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
            100 /var/lib/dpkg/status
    @edenhill
    Magnus Edenhill
    @edenhill
    whoa, that's very old
    Eliurkis
    @eliurkis_twitter
    It's the one that have Ubuntu on the repo
    Magnus Edenhill
    @edenhill
    You want 0.9.1 or later (0.9.3 was just pushed to latest ubuntu)
    it should be safer than adding a newer ubuntu repo
    Eliurkis
    @eliurkis_twitter
    @edenhill Awesome! It's working. Thank you so much!
    ksachdeva11
    @ksachdeva11

    After running
    php55-rdkafka
    command, what do I do next?

    Thanks!

    Im using mac
    Ana-Maria Lazar
    @anamarialazar
    @edenhill I am getting "Uncaught PHP Exception RdKafka\Exception No such configuration property: socket.blocking.max.ms". I am trying to improve the speed in an FPM env as described in arnaud-lb/php-rdkafka#101
    Magnus Edenhill
    @edenhill
    @anamarialazar Looks like an old librdkafka version. What version are you on?
    Ana-Maria Lazar
    @anamarialazar
    librdkafka1:
    Installed: 0.8.3-1ubuntu2
    Magnus Edenhill
    @edenhill
    That's really old. Upgrade to 0.9.2 or 0.9.3
    Ana-Maria Lazar
    @anamarialazar
    Thanks
    Craig Patrick
    @cpats007
    Hi all - anyone know the best way to reduce latency in PHP talking to Kafka? I'm currently running at around 30ms for just the process of writing data to Kafka - seems like a long time to me?
    Magnus Edenhill
    @edenhill
    @cpats007 use librdkafka 0.9.4 and set queue.buffering.max.ms to your liking
    Craig Patrick
    @cpats007

    thanks @edenhill - I've already set it to 1ms, but just going to upgrade to version 0.9.4 and see if that helps. Also, the way I was calling poll was with the following:

    $this->getProducer()->poll(-1);

    which from what I can gather waits until it gets a response. I tried changing to a while loop of $producer->poll(10); etc but didn't see any difference - so are there any implications in using my original methodology of $this->getProducer()->poll(-1); or any reasons not to do it that way etc?

    Magnus Edenhill
    @edenhill
    produce(); poll(-1); will not necessarily serve the delivery report of the message produced by that specific call to produce(). It may return sooner than that because of other callbacks (stats, error ,.etc)