Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Neil Young
    @nyoung
    yeah, i don't want the php request to sit and wait forever - i need it to drop the message on the floor and complete
    Craig Patrick
    @cpats007
    does this not work:
    $topicConfig = new TopicConf();
    $topicConfig->set('message.timeout.ms', 1000ms);
    Neil Young
    @nyoung
    trying it
    thought i had been through all the settings - that one seems obvious
    (derp)
    Craig Patrick
    @cpats007
    obviously you’ll need to change it for your code, bu then this:
    /** @var ProducerTopic $kafkaTopic */
    $kafkaTopic = $this->getProducer()->newTopic($topicName, $topicConfig);
    Neil Young
    @nyoung
    yeah
    Craig Patrick
    @cpats007
    :)
    Neil Young
    @nyoung
    i'm loading a topic config
    Craig Patrick
    @cpats007
    :thumbsup:
    Neil Young
    @nyoung
    Aw yeah, that worked - thanks @cpats007
    Craig Patrick
    @cpats007
    no problem
    Neil Young
    @nyoung
    @cpats007 did you have any luck with improving the performance on 1 or 2 kafka messages per instantiation?
    Craig Patrick
    @cpats007
    @nyoung I’ve got it to around somewhere between 10ms and 28ms in different apps writing data to Kafka with a single message - these are messages of around 1.5k (1500 bytes) so I guess they could be considered large?
    Mangoer
    @mangoer-ys
    when set 'offset.store.method' = 'broker' , where is the true place of broker on
    borker
    who know
    Magnus Edenhill
    @edenhill
    @mangoer-ys Offsets are written to the __consumer_offsets topic (by the broker)
    Mangoer
    @mangoer-ys
    @edenhill But I have a doubt that the last modification time of __consumer_offsets in log-file is not modified as a consumer commits offset
    and the size of __consumer_offsets file is 0.....
    Magnus Edenhill
    @edenhill
    @mangoer-ys that's a replicated topic with 50 (by default) partitions
    Mangoer
    @mangoer-ys
    I'm sorry. I mean the size of log file in __consumer_offsets-0 ~ 49 is 0
    Magnus Edenhill
    @edenhill
    okay. are you using manual or auto commits?
    and have you configured a group.id? (required for broker based commits)
    Mangoer
    @mangoer-ys
    Oh I know why. It's my error. I found a mistake that I only see the last modification time of consumer_offsets directory is not modified , but only one log file in 50 consumer_offset directory has been modified in fact.
    Thank you!
    Mangoer
    @mangoer-ys
    I found that using kafkaconsumer::commit it sends offset to broker every time as a consumer consume a message. So the log file is very big. How slove the issue?
    Magnus Edenhill
    @edenhill
    @mangoer-ys Use auto commits instead
    Mangoer
    @mangoer-ys
    :( must set manual commit in my business
    Magnus Edenhill
    @edenhill
    @mangoer-ys okay, manual commits are fine, albeit a bit slow, so you should typically avoid doing it per message, or at least do them asynchronously.
    Mangoer
    @mangoer-ys
    yes, I use commitasync. I expect to batching commit offsets.
    Joe Green
    @joegreen88
    I am using high level consumer for some period of time, and then I call unsubscribe(), but my php script doesn't terminate
    Magnus Edenhill
    @edenhill
    make sure to use librdkafka v0.9.4, it fixes a bunch of termination hang corner cases
    Neil Young
    @nyoung
    @edenhill do you have a timeline for when you are going to cut a tag off the latest master?
    Magnus Edenhill
    @edenhill
    @nyoung there will probably be a maintenance release this week, maybe not off master though but based on v0.9.5 + cherry-picks
    Neil Young
    @nyoung
    sounds good
    Joe Green
    @joegreen88
    I've upgraded librdkafka to v0.9.4 and my php process is still not terminating
    is there anything I need to do in addition to $consumer->unsubscribe() ?
    Joe Green
    @joegreen88
    Also tried reducing socket.blocking.max.ms
    Joe Green
    @joegreen88
    @arnaud-lb Have you encountered this issue before? PHP7, high level consumer, I have my loop set up to run for a certain amount of time, after the loop I call $consumer->unsubscribe(), then script ends. But php process is left hanging after consume loop is finished.
    Magnus Edenhill
    @edenhill
    @joegreen88 file an issue on php-rdkafka with a small reproducible test case
    Joe Green
    @joegreen88
    I've solved the issue, by unsetting the variable containing the RdKafka\KafkaConsumer instance after unsubscribing
    Not sure why that works but it does :thumbsup:
    flyleegit
    @flyleegit
    Hi,when I installed php-rdkafka,the errors were as follows,could anyone help me
    checking for rd_kafka_new in -lrdkafka... no
    configure: error: wrong rdkafka lib version or lib not found
    Magnus Edenhill
    @edenhill
    @flyleegit make sure librdkafka is installed before hand
    flyleegit
    @flyleegit
    yes,I have already installed
    flyleegit
    @flyleegit
    I have solved the issue, by export the LD_LIBRARY_PATH.Thanks
    COLE Edouard
    @sandvige
    Hello, we are handling tons of messages with php-rdkafka and librdkafka, and this is very good job guys. We're wondering if some of you already encountered this behaviour: when a consumer have to be properly stopped, we send a SIGINT to the PHP process, and we call pcntl_signal_dispatch() after each consumed message. We registered a callback on SIGINT, and simply tell the main loop to quit as soon as possible. This is working like a charm when the consumer have been started for a few minutes, but when the PHP process is running for... let say 10 days, it never finishes. It looks like it is waiting for things to finish, but this is never happening, and we have to kill it.
    We're using librdkafka (2213fb29f98a7a73f22da21ef85e0783f6fd67c4) and php-rdkafka (86feceba2469dd3442d96d0f73ea65c916b8f17f) and PHP 5.6.30