These are chat archives for arnaud-lb/php-rdkafka

24th
Jun 2016
Sébastien HOUZÉ
@shouze
Jun 24 2016 11:41
@arnaud-lb Hi, just few questions, some are probably just kafka only related (we're kafka recent adopters).
Arnaud Le Blanc
@arnaud-lb
Jun 24 2016 11:41
Hi
Sébastien HOUZÉ
@shouze
Jun 24 2016 11:42
  1. About consumers: When I see this example, you set a timeout of 120s & die early in your consumer (I guess it restart always). Is it better to do that or to have a long lived consumer process with shorter timeout (let's say about 1 to 5 seconds)?
Arnaud Le Blanc
@arnaud-lb
Jun 24 2016 11:45
If you are using the high level KafkaConsumer with automatic partition assignment, it’s better to have long lived consumer processes - this avoids negociation times when the consume re-joins
Sébastien HOUZÉ
@shouze
Jun 24 2016 11:45
2/ About both consumers & producers: is it better to let them create topics with their conf automatically? If yes, from consumers, from producers, both in competition? (we plan to deploy microservices running in docker containers)
Yes, using the high level one - so the long lived - breaking fast from the main loop & restarting quickly
Arnaud Le Blanc
@arnaud-lb
Jun 24 2016 11:47
Kafka makes it difficult to change some topic parameters, so I prefer to create them upfront with the right parameters. My production Kafka cluster has auto-creation disabled.
Sébastien HOUZÉ
@shouze
Jun 24 2016 11:50
Ok, I was mixed on this choice. Auto creation looks sexy at first glance for continuous deployment.
Arnaud Le Blanc
@arnaud-lb
Jun 24 2016 11:51
If default partitions count and replication factor works for most of your topics, auto creation could work for you
These are configurable, if you can find a configuration that works for all topics, go for auto creation