CODEWITHSUNDEEP
apache-kafka
Pinidbest
Pinidbest

Reputation: 4306

kafka consumer reads the same message

I have a single Topic with 5 partitions. I have 5 threads, each creating a Consumer All consumer are with the same consumer group using group.id. I also gave each consumer a different and unique client.id

I see that 2 consumers are reading the same message to process Should kafka handle this? How do I troubleshoot it?

Upvotes: 1

Views: 1397

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26865

Consumers within the same group should not receive the same messages. The partitions should be split across all consumers and at any time Kafka's consumer group logic ensures only 1 consumer is assigned to each partition.

The exception is if 1 consumer crashes before it's able to commit its offset. In that case, the new consumer that gets assigned the partition will re-consume from the last committed offset.

You can use the consumer group tool kafka-consumer-groups that comes with Kafka to check the partitions assigned to each consumer in your group.

Upvotes: 1

Related Questions