Kafka Broadcast Model for Producer-Consumer

Question

I have a topic A and 10 Different consumer groups (Lets say A1, A2, A3 ....A10). The consuming rate of each consumer group is different (Lets say 10 msg/s, 30 msg/s, 50 msg/s .......15 msg/s).

Each message is broadcast for each group. My A1 is slowest and A3 is fastest. Will each consumer group get all messages or there will be message drop as the different consuming rate.

Answer 1

Consumer Groups are independent of each other. Each consumer group will receive all messages from the topic; they will not drop messages.

Bear in mind your topic's retention configuration - if a consumer group drops so far behind that the data has expired from the topic then it would no longer be able to read those messages.

Refs:

• http://kafka.apache.org/documentation.html#intro_consumers
• http://kafka.apache.org/documentation.html#kafka_mq
• http://kafka.apache.org/documentation.html#basic_ops_consumer_lag