Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Mangoer
    @mangoer-ys
    when set 'offset.store.method' = 'broker' , where is the true place of broker on
    borker
    who know
    Magnus Edenhill
    @edenhill
    @mangoer-ys Offsets are written to the __consumer_offsets topic (by the broker)
    Mangoer
    @mangoer-ys
    @edenhill But I have a doubt that the last modification time of __consumer_offsets in log-file is not modified as a consumer commits offset
    and the size of __consumer_offsets file is 0.....
    Magnus Edenhill
    @edenhill
    @mangoer-ys that's a replicated topic with 50 (by default) partitions
    Mangoer
    @mangoer-ys
    I'm sorry. I mean the size of log file in __consumer_offsets-0 ~ 49 is 0
    Magnus Edenhill
    @edenhill
    okay. are you using manual or auto commits?
    and have you configured a group.id? (required for broker based commits)
    Mangoer
    @mangoer-ys
    Oh I know why. It's my error. I found a mistake that I only see the last modification time of consumer_offsets directory is not modified , but only one log file in 50 consumer_offset directory has been modified in fact.
    Thank you!
    Mangoer
    @mangoer-ys
    I found that using kafkaconsumer::commit it sends offset to broker every time as a consumer consume a message. So the log file is very big. How slove the issue?
    Magnus Edenhill
    @edenhill
    @mangoer-ys Use auto commits instead
    Mangoer
    @mangoer-ys
    :( must set manual commit in my business
    Magnus Edenhill
    @edenhill
    @mangoer-ys okay, manual commits are fine, albeit a bit slow, so you should typically avoid doing it per message, or at least do them asynchronously.
    Mangoer
    @mangoer-ys
    yes, I use commitasync. I expect to batching commit offsets.
    Joe Green
    @joegreen88
    I am using high level consumer for some period of time, and then I call unsubscribe(), but my php script doesn't terminate
    Magnus Edenhill
    @edenhill
    make sure to use librdkafka v0.9.4, it fixes a bunch of termination hang corner cases
    Neil Young
    @nyoung
    @edenhill do you have a timeline for when you are going to cut a tag off the latest master?
    Magnus Edenhill
    @edenhill
    @nyoung there will probably be a maintenance release this week, maybe not off master though but based on v0.9.5 + cherry-picks
    Neil Young
    @nyoung
    sounds good
    Joe Green
    @joegreen88
    I've upgraded librdkafka to v0.9.4 and my php process is still not terminating
    is there anything I need to do in addition to $consumer->unsubscribe() ?
    Joe Green
    @joegreen88
    Also tried reducing socket.blocking.max.ms
    Joe Green
    @joegreen88
    @arnaud-lb Have you encountered this issue before? PHP7, high level consumer, I have my loop set up to run for a certain amount of time, after the loop I call $consumer->unsubscribe(), then script ends. But php process is left hanging after consume loop is finished.
    Magnus Edenhill
    @edenhill
    @joegreen88 file an issue on php-rdkafka with a small reproducible test case
    Joe Green
    @joegreen88
    I've solved the issue, by unsetting the variable containing the RdKafka\KafkaConsumer instance after unsubscribing
    Not sure why that works but it does :thumbsup:
    flyleegit
    @flyleegit
    Hi,when I installed php-rdkafka,the errors were as follows,could anyone help me
    checking for rd_kafka_new in -lrdkafka... no
    configure: error: wrong rdkafka lib version or lib not found
    Magnus Edenhill
    @edenhill
    @flyleegit make sure librdkafka is installed before hand
    flyleegit
    @flyleegit
    yes,I have already installed
    flyleegit
    @flyleegit
    I have solved the issue, by export the LD_LIBRARY_PATH.Thanks
    COLE Edouard
    @sandvige
    Hello, we are handling tons of messages with php-rdkafka and librdkafka, and this is very good job guys. We're wondering if some of you already encountered this behaviour: when a consumer have to be properly stopped, we send a SIGINT to the PHP process, and we call pcntl_signal_dispatch() after each consumed message. We registered a callback on SIGINT, and simply tell the main loop to quit as soon as possible. This is working like a charm when the consumer have been started for a few minutes, but when the PHP process is running for... let say 10 days, it never finishes. It looks like it is waiting for things to finish, but this is never happening, and we have to kill it.
    We're using librdkafka (2213fb29f98a7a73f22da21ef85e0783f6fd67c4) and php-rdkafka (86feceba2469dd3442d96d0f73ea65c916b8f17f) and PHP 5.6.30
    (and kafka version: kafka_2.11-0.10.0.1)
    Magnus Edenhill
    @edenhill
    @sandvige 0.9.1 is really old and the consumer has seen a lot of fixes since then. I suggest you try librdkafka v0.9.5 (or 0.11.0 which will be released soon)
    COLE Edouard
    @sandvige
    @edenhill Thanks!
    Do you maintain somewhere an upgrade guide to know what changed since? We noticed the TopicConf class have been moved
    (we're investigating as it looks this class is not found, but i'm wondering why.. it looks more like a configuration issue)
    Magnus Edenhill
    @edenhill
    https://github.com/edenhill/librdkafka/releases
    COLE Edouard
    @sandvige
    I was talking about the php classes, sorry :). Thanks!
    Richard
    @computerichy

    Hello,

    Since restarting our web servers, librdkafka is failing to initialise. It's possible that PHP has been updated, but nevertheless it has been running PHP 7 throughout, and was working at one time. This is the error being logged in the apache error_logs:

    PHP Warning: PHP Startup: rdkafka: Unable to initialize module\nModule compiled with build ID=API20151012,NTS\nPHP compiled with build ID=API20151012,TS\nThese options need to match\n in Unknown on line 0

    Here's the current 'php -v' output:

    PHP 7.0.18 (cli) (built: Apr 11 2017 14:25:57) ( NTS )
    Copyright (c) 1997-2017 The PHP Group
    Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.18, Copyright (c) 1999-2017, by Zend Technologies

    Magnus Edenhill
    @edenhill
    @computerichy looks like the module was compiled for another PHP version or with other flags. Try rebuilding it
    Richard
    @computerichy
    @edenhill Thanks Magnus, the module was installed via PECL. Running 'pecl list' prints this 'rdkafka 3.0.3 stable'.
    Is it supposed to be marked as non-thread safe?
    Magnus Edenhill
    @edenhill
    @computerichy librdkafka is thread-safe, I have no idea if the php module is, I've never used it :|
    Richard
    @computerichy
    @edenhill Ahhh, turns out the PHP module isn't. The solution was to switch back from the Worker MPM to Prefork. Thanks!
    remizyaka
    @remizyaka
    Hi. is it possible to commit offset manually when using low level consumer? Consumer::offsetStore for some reason isn't updating storred offset after offset automatically reset