Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Magnus Edenhill
    @edenhill
    Nope
    axiao
    @joostshao
    https://arnaud-lb.github.io/php-rdkafka/phpdoc/rdkafka-producertopic.produce.html how to define the producer send a message successfully or failed ?
    the doc said no return , just null
    Arnaud Le Blanc
    @arnaud-lb
    @joostshao: produce is asynchronous, but you can set a delivery callback with RdKafka\Conf::setDrMsgCb(). This will allow you to be notified when a message delivery fails or succeeds
    Ferran Caellas Puig
    @FerCa
    Hi again! Any plan on releasing the high level consumer?
    Arnaud Le Blanc
    @arnaud-lb
    Hi @FerCa ! I’ll try to release it soon. In the meantime, you can install it manually: https://arnaud-lb.github.io/php-rdkafka/phpdoc/rdkafka.installation.manual.html
    Ferran Caellas Puig
    @FerCa
    ok! thanks!
    Ferran Caellas Puig
    @FerCa
    I'm having problems using the low level and consumer groups
    and I'm setting the group id in the conf for the consumer:
    $conf->set('group.id', $config['consumerGroup']);
    I have 3 partitions in the topic
    and I'm starting 2 consumers consuming the 3 partitions
    since my two consumers are in the same consumer group, I was expecting the messages to be distributed between my 2 consumers
    but I'm getting all the messages in the 2 consumers
    Arnaud Le Blanc
    @arnaud-lb
    In the low-level consumer, you choose which partitions you are consuming. If each consumer consume different partitions, you get different messages in each consumer. Else, you can the same messages in each consumer :)
    Ferran Caellas Puig
    @FerCa
    ok, then here the consumer groups didn't apply?
    I mean, its irrellevant to which consumer group a consumer belongs?
    Arnaud Le Blanc
    @arnaud-lb
    In the low level consumer, the consumer group is used only as a namespace when storing offsets.
    Ferran Caellas Puig
    @FerCa
    ok, thanks for the clarification!
    Arnaud Le Blanc
    @arnaud-lb
    If you want automatic balancing of partitions accross consumers, you should be using the high level consumer
    axiao
    @joostshao
    %3|1467795146.244|FAIL|rdkafka#consumer-1| 172.16.0.148:9098/bootstrap: Connect to ipv4#172.16.0.148:9098 failed: Connection refused
    %3|1467795146.244|ERROR|rdkafka#consumer-1| 172.16.0.148:9098/bootstrap: Connect to ipv4#172.16.0.148:9098 failed: Connection refused
    %3|1467795146.244|ERROR|rdkafka#consumer-1| 1/1 brokers are down
    0
    "Success"
    -196
    why can not get error info when all brokers are down ?
    axiao
    @joostshao
    or the error info has no difference , when brokers all down or all avaliable
    ahuigo
    @ahuigo
    Hi, I come across an error : "%3|1467888104.894|FAIL|rdkafka#consumer-1| 10.160.133.7:9092/3: Receive failed: Disconnected", where I try to connect kafka.
    This message was deleted
    ahuigo
    @ahuigo
    $rk = new RdKafka\Producer();
    $rk->setLogLevel(LOG_DEBUG);
    $rtn = $rk->addBrokers("10.160.132.242:2181");
    var_dump($rtn);

    output:

    %3|1467888397.800|FAIL|rdkafka#producer-1| 10.160.132.242:2181/bootstrap: Receive failed: Disconnected

    Can I replace 10.160.132.242:2181/bootstrap with 10.160.132.242:2181/kafka-sys-lycc?

    This message was deleted
    Magnus Edenhill
    @edenhill
    @hilojack the /bootstrap part is appended by librdkafka to differntiate it from a automatically broker learnt from cluster metadata (which then has the broker id after the slash, e.g,. ../3 )
    so no, it cant be changed.
    ahuigo
    @ahuigo
    @edenhill ok
    ahuigo
    @ahuigo

    Is where any way to get the number of partitions of a specified topic?

    $topic =  $kafkaConsumer->newTopic($topicName, $topicConf);
    $partitions_num = $topic->getPartitionsNum();// any way??
    for($i=0; $i<$partitions_num, $i++){
      $topic->consumeStart($i, RD_KAFKA_OFFSET_BEGINNING);
    
    }

    @edenhill

    Magnus Edenhill
    @edenhill
    Use the metadata api
    ahuigo
    @ahuigo
    Thanks, I'll have a try.
    ahuigo
    @ahuigo
    I found that my php-kafka is blocked by futex, anybody knows about this problem?
    22:05:19 futex(0x7fc3e1177954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1468247119, 308563000}, ffffffff
    axiao
    @joostshao
    is RD_KAFKA_PARTITION_UA just used for producer ?
    how to consumer kafka topic from all partitions?
    axiao
    @joostshao
    %3|1468558926.735|FAIL|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558926.735|ERROR|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558926.735|ERROR|rdkafka#consumer-1| 1/1 brokers are down
    %3|1468558926.847|FAIL|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558926.848|ERROR|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558926.848|ERROR|rdkafka#consumer-1| 1/1 brokers are down
    %3|1468558927.441|FAIL|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558927.441|ERROR|rdkafka#consumer-1| 172.16.0.148:9092/0: Receive failed: Disconnected
    %3|1468558927.442|ERROR|rdkafka#consumer-1| 1/1 brokers are down
    when i use $conf->setRebalanceCb
    ahuigo
    @ahuigo
    @joostshao
    1. RD_KAFKA_PARTITION_UA is just used for producer, because it lets librdkafka choose the partition.
    2. how to consumer kafka topic from all partitions?
      You could get these partitions via Metadata:
           $topic = $consumer->newTopic("topic name", $topicConf);
           $metaData = $consumer->getMetadata(false, $topic, 1000);
           $partitions = $metaData->getTopics()->current()->getPartitions();
    axiao
    @joostshao
    @hilojack thank you, i am trying
    i am confusing about the pramaters
    When TRUE, request info about all topics in cluster. Else, only request info about locally known topics.
    what for locally known topics meaning ?
    axiao
    @joostshao
    RdKafka\Metadata\Collection {#7}
    RdKafka\Metadata {#6}
    the weekend
    axiao
    @joostshao
    sorry, may be i found the prolbem, high level need the server upgrade to 0.9.0.x too, otherwise the protocol will be missing match .