Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Neil Young
    @nyoung
    heya folks - trying to use the php-rdkafka tool and it seems to be working well for our use case, except I can't figure out how to kill a publish or setup a publish timeout if there is some kind of connection error on all brokers
    it just seems to sit and spin for a long time for me - is there some way to force it to fail from PHP?
    Craig Patrick
    @cpats007
    what is the issue you are having, so if the producer can’t produce, you want it to fail in PHP?
    Neil Young
    @nyoung
    yeah, i don't want the php request to sit and wait forever - i need it to drop the message on the floor and complete
    Craig Patrick
    @cpats007
    does this not work:
    $topicConfig = new TopicConf();
    $topicConfig->set('message.timeout.ms', 1000ms);
    Neil Young
    @nyoung
    trying it
    thought i had been through all the settings - that one seems obvious
    (derp)
    Craig Patrick
    @cpats007
    obviously you’ll need to change it for your code, bu then this:
    /** @var ProducerTopic $kafkaTopic */
    $kafkaTopic = $this->getProducer()->newTopic($topicName, $topicConfig);
    Neil Young
    @nyoung
    yeah
    Craig Patrick
    @cpats007
    :)
    Neil Young
    @nyoung
    i'm loading a topic config
    Craig Patrick
    @cpats007
    :thumbsup:
    Neil Young
    @nyoung
    Aw yeah, that worked - thanks @cpats007
    Craig Patrick
    @cpats007
    no problem
    Neil Young
    @nyoung
    @cpats007 did you have any luck with improving the performance on 1 or 2 kafka messages per instantiation?
    Craig Patrick
    @cpats007
    @nyoung I’ve got it to around somewhere between 10ms and 28ms in different apps writing data to Kafka with a single message - these are messages of around 1.5k (1500 bytes) so I guess they could be considered large?
    Mangoer
    @mangoer-ys
    when set 'offset.store.method' = 'broker' , where is the true place of broker on
    borker
    who know
    Magnus Edenhill
    @edenhill
    @mangoer-ys Offsets are written to the __consumer_offsets topic (by the broker)
    Mangoer
    @mangoer-ys
    @edenhill But I have a doubt that the last modification time of __consumer_offsets in log-file is not modified as a consumer commits offset
    and the size of __consumer_offsets file is 0.....
    Magnus Edenhill
    @edenhill
    @mangoer-ys that's a replicated topic with 50 (by default) partitions
    Mangoer
    @mangoer-ys
    I'm sorry. I mean the size of log file in __consumer_offsets-0 ~ 49 is 0
    Magnus Edenhill
    @edenhill
    okay. are you using manual or auto commits?
    and have you configured a group.id? (required for broker based commits)
    Mangoer
    @mangoer-ys
    Oh I know why. It's my error. I found a mistake that I only see the last modification time of consumer_offsets directory is not modified , but only one log file in 50 consumer_offset directory has been modified in fact.
    Thank you!
    Mangoer
    @mangoer-ys
    I found that using kafkaconsumer::commit it sends offset to broker every time as a consumer consume a message. So the log file is very big. How slove the issue?
    Magnus Edenhill
    @edenhill
    @mangoer-ys Use auto commits instead
    Mangoer
    @mangoer-ys
    :( must set manual commit in my business
    Magnus Edenhill
    @edenhill
    @mangoer-ys okay, manual commits are fine, albeit a bit slow, so you should typically avoid doing it per message, or at least do them asynchronously.
    Mangoer
    @mangoer-ys
    yes, I use commitasync. I expect to batching commit offsets.
    Joe Green
    @joegreen88
    I am using high level consumer for some period of time, and then I call unsubscribe(), but my php script doesn't terminate
    Magnus Edenhill
    @edenhill
    make sure to use librdkafka v0.9.4, it fixes a bunch of termination hang corner cases
    Neil Young
    @nyoung
    @edenhill do you have a timeline for when you are going to cut a tag off the latest master?
    Magnus Edenhill
    @edenhill
    @nyoung there will probably be a maintenance release this week, maybe not off master though but based on v0.9.5 + cherry-picks
    Neil Young
    @nyoung
    sounds good
    Joe Green
    @joegreen88
    I've upgraded librdkafka to v0.9.4 and my php process is still not terminating
    is there anything I need to do in addition to $consumer->unsubscribe() ?
    Joe Green
    @joegreen88
    Also tried reducing socket.blocking.max.ms
    Joe Green
    @joegreen88
    @arnaud-lb Have you encountered this issue before? PHP7, high level consumer, I have my loop set up to run for a certain amount of time, after the loop I call $consumer->unsubscribe(), then script ends. But php process is left hanging after consume loop is finished.
    Magnus Edenhill
    @edenhill
    @joegreen88 file an issue on php-rdkafka with a small reproducible test case
    Joe Green
    @joegreen88
    I've solved the issue, by unsetting the variable containing the RdKafka\KafkaConsumer instance after unsubscribing
    Not sure why that works but it does :thumbsup:
    flyleegit
    @flyleegit
    Hi,when I installed php-rdkafka,the errors were as follows,could anyone help me
    checking for rd_kafka_new in -lrdkafka... no
    configure: error: wrong rdkafka lib version or lib not found
    Magnus Edenhill
    @edenhill
    @flyleegit make sure librdkafka is installed before hand
    flyleegit
    @flyleegit
    yes,I have already installed