Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    axiao
    @joostshao
    php —ri | grep rdkafka @juise
    this is just available for librdkafka >= 0.9
    just for high level consumer
    Alexander Petrovsky
    @juise
    @joostshao hmmm,
    $ php —ri | grep rdkafka
    $
    what should I do?
    why
    $ php -i | grep extension_dir
    extension_dir => /usr/lib/php5/20121212 => /usr/lib/php5/20121212
    so, why the command above shows me that extension dir is exists and used?
    ahuigo
    @ahuigo
    @juise Have you loaded rdkafka.so in php.ini ? If not, config it with shell echo "extension=rdkafka.so" >> $(php --ini |grep -oP 'Scan for additional .ini files in: \S+' | grep -oP '/\S+')/rdkafka.ini
    ahuigo
    @ahuigo
    I found that php-rdkafka script will hung out if there is any uncaught exception.
    ahuigo
    @ahuigo
    More detail: PHP 5.3.29 (cli) will coredump if there is any uncaught exception.
    Syed Salman Ahmed Bukhari
    @salmanbukhari
    Is RDKafkaProducer thread safe ? I want to put just one message in kafka topic for each web request. I have around 1000 requests per second. So I think recreating connection for each request will not be a good idea. Can I use Producer as shared object?
    Arnaud Le Blanc
    @arnaud-lb
    Hi @salmanbukhari
    Syed Salman Ahmed Bukhari
    @salmanbukhari
    @arnaud-lb hello
    Arnaud Le Blanc
    @arnaud-lb
    The underlying producer is thread safe, but the PHP wrapper probably not. Are you using the pthread extension ?
    Syed Salman Ahmed Bukhari
    @salmanbukhari
    No actually I am doing it in main thread. I just need one message per PHP page request. But I have found that it is not possible as PHP forks a new process for every request.So a new connection is created every time to push a message in Kafka. It is taking 400ms per request. I think I need to revise my model.
    axiao
    @joostshao
    hi, anyone still online ?
    how to get the result of producer failed? it is blocking for 3 seconds
    Vladislav Krakhalev
    @vchampion
    Hi! Arnauld is it possible to get offset which is assigned to successfull sended message in Kafka?
    I mean directrly when produce is called
    Magnus Edenhill
    @edenhill
    librdkafka will return the produced offset if you pass it produce.offset.report=true (topic config). But I'm not sure the offset is actually exposed in PHP.
    The offset will be available in the delivery report
    Vladislav Krakhalev
    @vchampion

    Thank you, Magnus! I'll check how it will work with current version.

    Now in version 0.9.1 of php-rdkafka I didn't found methods to obtain offset.. In manual is setDrMsgCb, but it exists only in master branch

    Magnus Edenhill
    @edenhill
    Yeah, looks like setDrMsgCb should work
    nEAnnam
    @nEAnnam
    Hi, i've tried to set the delivery callback (setDrMsgCb), but it is not executed. Default configurations and i've polled while getOutQLen > 0, Did I miss something?
    nEAnnam
    @nEAnnam
    $conf = new \RdKafka\Conf();
    $conf->setDrMsgCb(function($kafka, $message) {
        echo $message->err;
    });
    
    $rk = new \RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rk->addBrokers("localhost:9092");
    
    $conf = new \RdKafka\TopicConf();
    $conf->set('request.required.acks', -1);
    $conf->set('produce.offset.report', true);
    $topic = $rk->newTopic("test", $conf);
    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message payload");
    
    $rk->poll(100);
    while($rk->getOutQLen())
                $rk->poll(100);
    Magnus Edenhill
    @edenhill
    @nEAnnam: You dont seem to pass the first $conf to Producer() ctor
    nEAnnam
    @nEAnnam
    OMG, lol
    sorry
    now works
    thank you
    cslinuxboy
    @cslinuxboy

    I'm having issues where my php producer script is consuming large amounts of memory and eventually causing swap issues. I'm using the high-level producer. Any ideas as to what I'm doing wrong? Code is below.
    '''
    $kafka_producer_obj = new RdKafka\Producer();

    if(!is_object($kafka_producer_obj))
    {
    syslog(LOG_CRIT, $this_file.": CRITICAL: Could not create a Kafka Producer object. Exiting now.");
    exit(1);
    }

    if($kafka_producer_obj->addBrokers(gethostname()) <= 0)
    {
    syslog(LOG_CRIT, $this_file.": CRITICAL: Could not add a broker to the Kafa producer object. Exiting now.");
    exit(1);
    }

    $kafka_producer_topic_obj = $kafka_producer_obj->newTopic(SonusDbAbstract::getRegistryValue("kafka_topic"));
    if(!is_object($kafka_producer_topic_obj))
    {
    syslog(LOG_CRIT, $this_file.": CRITICAL: Could not create a Kafka producer topic object. Exiting now.");
    exit(1);
    }

    ... for each line of data...
    $kafka_producer_topic_obj->produce(RD_KAFKA_PARTITION_UA, 0, $this_data_line);
    '''

    Sorry about the incorrect code block format. used ''' instead of ``` ;(
    Matthew Goslett
    @matthewgoslett
    Hi Everyone. I was wondering if there's any way to get the \RdKafka\Conf object after a \RdKafka\KafkaConsumer has been instantiated.
    I need to be able to change a consumer group id after the fact
    Magnus Edenhill
    @edenhill
    config cannot be changed after client creation
    Matthew Goslett
    @matthewgoslett
    ah ok, thanks
    Matthew Goslett
    @matthewgoslett
    also, it seems like the producer & consumer try to talk to the broker as soon as they're created. Is there anyway to stop that?
    It's an issue for me with Laravel specifically, when I composer update a package, and this is followed by a 'optimise' which combines all classes into a single compiled.php

    eg: Generating autoload files

    Illuminate\Foundation\ComposerScripts::postUpdate
    php artisan optimize
    %3|1473321097.324|FAIL|rdkafka#consumer-1| 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused
    %3|1473321097.324|ERROR|rdkafka#consumer-1| 127.0.0.1:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused
    %3|1473321097.324|ERROR|rdkafka#consumer-1| 1/1 brokers are down
    Generating optimized class loader

    Magnus Edenhill
    @edenhill
    @matthewgoslett: You could call addBrokers() later when you want to get started
    just make sure not to specify any bootstrap.servers/metadata.broker.list properties
    Matthew Goslett
    @matthewgoslett
    ah ok got it, easy enough for the addBrokers() but for $conf->set('metadata.broker.list', $config['brokers']), my high level consumer is already created
    I know you said conf can't be changed after creation, but is their way to change the metadata.broker.list on the consumer?
    Magnus Edenhill
    @edenhill
    Nope.
    Efimov Evgenij
    @edefimov
    Hi everyone! I have a problem sending message in php7. I'm producing message via cli, the message is successfully published into kafka. When the script is about to terminate the RdKafka\Producer::__destruct method is called and hangs forever, so the cli process never dies. Has anyone encountered with such a behaviour?
    Magnus Edenhill
    @edenhill
    @edefimov: do you know which underlying librdkafka version you are using?
    Efimov Evgenij
    @edefimov
    @edenhill , 0.9.1
    Magnus Edenhill
    @edenhill
    okay, there are some fixes on master for termination hangs
    and there will be a new formal release in about a month
    Efimov Evgenij
    @edefimov
    thanks!
    sbscrape
    @sbscrape
    hi, just been experimenting with this, but I cant find any reference to how to do offsetRequest-s, whereas I don't want to store the offset but ask for it based on a timestamp, is that currentyl possible at all?