These are chat archives for arnaud-lb/php-rdkafka

19th
Oct 2018
amnai2p
@amnai2p
Oct 19 2018 12:36

Hi everyone
I am trying to create a topic test which should have following configurations
cleanup.policy = compact and retention.ms = 86400000(milliseconds)
RdKafka\TopicConf() object wont set these for a topic (Throwing exception )
here is my code

$rk = new RdKafka\Producer();
$rk->setLogLevel(LOG_DEBUG);
$rk->addBrokers("127.0.0.1");

// Set the topic configuration:
$topicConf = new RdKafka\TopicConf();
//$topicConf->set("auto.commit.interval.ms", 1e3);
$topicConf->set("cleanup.policy", "compact");
$topicConf->set("retention.ms", 86400000);

$topic = $rk->newTopic("test", $topicConf);
$message = "Meassge testing topic config";
try {
    $topic->produce(RD_KAFKA_PARTITION_UA, 0, $message);
    // We have to call the poll function to make sure everything did as it should
   // $rk->poll(-1);
} catch (\Exception $exception) {
    //something went wrong, so handle your errors as you would...
    return false;
}

However I can create the topic with same configuration

kafka-topics --create --zookeeper localhost:2181 \--replication-factor 1 --partitions 1 --topic test --config retention.ms=86400000 --config cleanup.policy=compact

Either I am doing it wrong in php or there is any other way to set these configuration
Any help will be really appreciated

karavzeka
@karavzeka
Oct 19 2018 13:08
You confused kafka flags and librdkafka flags.
https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - this is the list of possible flags for library
amnai2p
@amnai2p
Oct 19 2018 13:09
Yes I have seen that list and these are not there
How can I set these flags ? from librdkafka
karavzeka
@karavzeka
Oct 19 2018 13:11
I think it's impossible
Set topic properties in kafka config.
amnai2p
@amnai2p
Oct 19 2018 13:14
Yeah seems like that is the only solution
alright. Thank you for your feedback!
karavzeka
@karavzeka
Oct 19 2018 13:14
php-rdkafka - it's not a library for managing kafka topics.
It's sad that there isn't convenient api for managing topics. Only kafka-topics shell script.
amnai2p
@amnai2p
Oct 19 2018 13:18
yeah even REST Proxy api does not have anything related to topic creation