by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Paweł Niedzielski
    @steveb_gitlab
    You can always increase the number of partitions in topic (which will cause Kafka to rebalance though)
    Brendan
    @balexander
    ok I think that makes sense. Thanks again
    Paweł Niedzielski
    @steveb_gitlab
    In case of my application as an example I'm having a topic with 20 partitions shared between 4-5 consumers
    Just keep in mind that due to partitioning messages might arrive in different order
    i.e. later arriving message from partition 1 can be processed sooner than earlier message from partition 2
    You can check number of partitions from RdKafka\Metadata\Topic::getPartitions()
    Amna Mazhar
    @amnamazhar
    Hi @steveb_gitlab
    do you know when this feature will be in production " Implement librdkafka Topic API #215" ?
    Paweł Niedzielski
    @steveb_gitlab
    @amnamazhar I was busy with other things unrelated to Kafka at work. I'll try to have a crack on it at the weekend, but can't promise anything.
    Amna Mazhar
    @amnamazhar
    @steveb_gitlab Thanks for considering it!
    Paweł Niedzielski
    @steveb_gitlab
    @amnamazhar I've taken a look at how it should work, but it will take some time, especially since I'm not really a specialist in C programming ;)
    Amna Mazhar
    @amnamazhar
    @steveb_gitlab thank you for looking into that! Really appreciate it! If you find anything useful let me know!
    I can use kafka proxy rest api to get that information but the client dont want that so probably make my own script for that or checkout the source code for librfkafka to enhance it myself
    Paweł Niedzielski
    @steveb_gitlab
    @amnamazhar librdkafka has all the capabilities needed. It's only code in phprdkafka that is missing. There are examples I could point you to in librdkafka if you want to :) maybe you can give it a shot yourself :)
    In phprdkafka, queues functions are already present in queue.c
    Amna Mazhar
    @amnamazhar

    @amnamazhar librdkafka has all the capabilities needed. It's only code in phprdkafka that is missing. There are examples I could point you to in librdkafka if you want to :) maybe you can give it a shot yourself :)

    @steveb_gitlab I will checkout the test sample you mentioned. if you can point me to any other example that would be great!

    Brendan
    @balexander
    I have a high-level consumer that just prints out the id of some test events I produced. I'm running that script twice (i.e., one consumer running in one terminal window and the other in a separate window) but only one consumes events at a time. Both consumers are in the same consumer group, and I have verified that the topic I'm consuming has 2 partitions, both of which have data. Am I wrong in thinking that both should be processing events concurrently? I haven't tried with a low level consumer yet. All configurations options are default except for the group id and some timeouts.
    Paweł Niedzielski
    @steveb_gitlab
    @balexander both should be consuming concurrently, but only after Kafka rebalances consumers
    first consumer that starts is assigned both partitions
    and only after consumer rebalancing (which should occur after a couple seconds usually) second consumer will be assigned the other partition and will start consuming
    until it happens, it will wait for assignments
    Brendan
    @balexander
    Yeah thats the behavior I was expecting. Weird.
    Brendan
    @balexander
    FYI the consumer problem I was having resolved itself. Not sure if it was some weird intermittent issue with my kafka set up, but creating a new topic with 2 partitions and running my consumers against that gave expected results
    aand18
    @aand18
    Hi! Is php-rdkafka comatible with PHP 7.2 and 7.3? I can't find DLL builds for the newer PHP version. Thanks
    Paweł Niedzielski
    @steveb_gitlab
    @aand18 while phprdkafka is compatible with PHP 7.2/7.3 (no major internal API changes in PHP that would cause issues) windows builds might not be created and published automatically.
    aand18
    @aand18
    @steveb_gitlab any clue as to why only v7.1 builds are created? Here are only empty 'logs' of v7.2 and v7.3 https://windows.php.net/downloads/pecl/releases/rdkafka/3.1.0/logs/
    Paweł Niedzielski
    @steveb_gitlab
    @aand18 probably the person that uploaded the builds no longer did so :) It might have not been automatic
    aand18
    @aand18
    @steveb_gitlab do you know of a guide for compiling your own DLLs? thanks
    Paweł Niedzielski
    @steveb_gitlab
    I don't have a windows machine at hand to compile, so I can't do that for you, and my net connection is bad (on mobile in a black hole :) )
    I did compile phprdkafka and other extension on linux, but never did on windows
    I'll try to dig a little
    @aand18 try it and see if it works - as I said, I can't check it instead of you since I don't have a win machine atm
    aand18
    @aand18
    @steveb_gitlab thanks, that's really kind of you. I'll take a shot at it. I hope it doesn't transform in a week of work, these kind of activities have a bad habit of doing so ...
    Paweł Niedzielski
    @steveb_gitlab
    yep, they do :D
    I don't really know if there was an automated push for windows builds
    hopefully Kafka will match your use-case and you'll stick with it ;)
    aand18
    @aand18
    I'm using for a while version 3.0.5, only for a consuming though. There's a bug that, when commanding the BAT script to close, the host console exits but the PHP process stays. So it's difficult to control through scheduled tasks. I was hoping version 3.1.0 fixes that.
    Paweł Niedzielski
    @steveb_gitlab
    @aand18 I'm not sure it's related to phprdkafka extension
    I mean, there is one issue that might cause librdkafka threads to stick around
    and that's if signals are not propagated properly
    It might be what you're looking for
    aand18
    @aand18
    @steveb_gitlab I don't think it works on Windows - signals in general I mean
    Paweł Niedzielski
    @steveb_gitlab
    I'm not sure as well
    technically they should work
    You're sure that the process is still present?
    Not just that Kafka does not reassign partitions to a new consumer?