These are chat archives for arnaud-lb/php-rdkafka

2nd
Jun 2016
Vladislav Krakhalev
@vchampion
Jun 02 2016 04:20
hello guys)
I have curious situation - when 2 consumers fetches messages from one partition with the same group really only one consumer have ability to fetch messages, another can't and raises timeout error
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 12:13
Hi Vladislav
Are you consuming the same set of partitions in the 2 consumers ?
Vladislav Krakhalev
@vchampion
Jun 02 2016 12:14
Yes, the same partitions
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 12:16
I think this is not well supported; usually different consumers consume from different partitions
Are you trying to consume all messages twice, or to consume half the messages in each consumer ?
Vladislav Krakhalev
@vchampion
Jun 02 2016 12:18
Mm.. yes half the messages as it on simple queue system
One message to one consumer another to the two
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 12:18
This could work only if the two consumers synchronized their offsets in some way; however, neither Kafkfa nor the client are trying to do so
You must use two different partitions
Partitions are Kafka’s way to distribute the load among consumers
Vladislav Krakhalev
@vchampion
Jun 02 2016 12:20
so one group one consumer?
As i remember kafka manual it's allowed to consume one partition by multiple consumers
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 12:22
One group can have multiple consumers, but high level consumers implementations will assign each consumer to a different partition
It’s allowed to consume one partition by multiple consumers as long as you want to consume the same messages in each consumer, and that you store consumer offsets in a way that consumers do not override each other’s offsets. E.g. if each consumer has a different group.id, it will work.
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:11
Yes, when consumer groups is different it's ok. But
from manual
"If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers."
So I understand you but in manual it's different
Or I incorrectly understand it?
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:15
From this phrase alone it seems possible that Kafka is going to balance messages from the same partition to different consumers, yes
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:15
However, I believe that the manual also explains that balancing is done by assigning different partitions to different consumers
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:15
hm
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:16
"By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group."
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:19
Good, ok, you are right
is it possible to automatic selection of partitions between consumers per consumer group?
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:21
(requires Kafka 0.9)
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:24
Thank you very much!) have a good day!)
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:24
You’re welcome :) Have a good day
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:42
Arnauld and one little question)
What do you think is it good idea to use kafka as replacement for rabbitmq with work queue?
I think it has some benefits and limitations
I want to make ESB in our company, but system administrator tell me what will be good to use rabbitmq )
Arnaud Le Blanc
@arnaud-lb
Jun 02 2016 13:51
RabbitMQ is quite good for relatively small workloads. Kafka scales well and it reliable at a large amount/rate of messages.
I replaced RabbitMQ completely by Kafka after it crashed under load, and after I failed to scale it to handle more load.
I think that RabbitMQ is still better for small workloads, though. It’s simpler to use and has more features.
Vladislav Krakhalev
@vchampion
Jun 02 2016 13:53
I have the same thoughts ) thank you