CODEWITHSUNDEEP
apache-kafkaspring-kafka
Marcelo Canaparro
Marcelo Canaparro

Reputation: 148

If my service consumes Kafka messages, can kafka somehow lose my offsets?

If I have a service that connects to kafka as a message consumer, and every message I read I send a commit to that message offset, so that if my service shutsdown and restarts it will start reading from the last read message onwards. My understanding is that the committed offset will be maintained by kafka.

Now my question is, do I have to worry about the offset? Can kafka somehow lose that information and when the service restarts start reading messages from the beginning of the topic or the end of it depending on my initial offset config? Or if kafka loses my offset it will also have lost all messages in the topic so that it is alright to read from the beginning?

Note: I use spring-kafka on the service, but not sure if that is relevant to the question.

Upvotes: 0

Views: 815

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191864

In most cases where you have an active consumer (with manual or auto-committing), you don't need to worry about it.

The cases where you do need to consider the behavior of auto.offset.reset setting is when the offsets.retention.minutes time on the broker has elapsed while your consumer group(s) are inactive. When this happens, Kafka compacts the __consumer_offsets topic and removes any offsets stored for those inactive groups

Losing offsets doesn't affect the source topic. Your client topic(s) have their own independent retention settings, and its message can be removed as well (or not), depending on how you've configured it.

Upvotes: 2

Related Questions