CODEWITHSUNDEEP
goapache-kafkaproducer-consumer
doubt
doubt

Reputation: 315

Kafka consumers picking up the same message again

I have a golang service with 30 instances. There is a topic with 3 partitions and a single consumer group listening to the messages. The issue we are facing is that each message is being picked up by the same consumer again and again with a gap of 24-30 hours. I have confirmed that the message is being pushed to the topic once and even the message is same every time but the message is being picked up at different time intervals, e.g.:

Can someone please help with what the problem could be. Thanks

Upvotes: 0

Views: 1708

Answers (1)

nipuna
nipuna

Reputation: 4095

  1. You don't need 30 instances to consume from 3 partitions. Because partitions are not shared between consumer instances. So, if your topic has 3 partitions, your maximum number of consumer instances should be 3.

  2. Re-Consuming can be happen if your consumers not commit the read massage after consumed. Look in to consumer commit interval and auto commit enabled or disabled. If your consumed message is not committed, then after rebalance happened to consumers, those messages can be re-consumed.

Upvotes: 3

Related Questions