CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Vaibhav Maniar
Vaibhav Maniar

Reputation: 210

Kafka: Who maintains that upto which offset number message is read by a consumer group?

I know that all the messages (or offset) in a Kafka Queue Partition has its offset number and it takes care of the sequence of offsets. But if I have a Kafka Consumer Group (or single Kafka Consumer) which is reading particularly the Kafka Topic Partition then how it maintains up to which offset messages are read and who maintains this offset counter? If the consumer goes down then how a new consumer will start reading the offset from the next unread (or not acknowledged) offset.

Upvotes: 1

Views: 557

Answers (1)

Michael Heil
Michael Heil

Reputation: 18475

The information about Consumer Groups is all stored in the internal Kafka topic __consumer_offsets. Whenever a new group tries to read data from a topic it checks its offset position in that internal topic which has a deletion policy set to compact. The compaction keeps this topic small.

Kafka comes with a command line tool kafka-consumer-groups.sh that helps you understand which information is stored for each consumer group.

More information is given in the Kafka Documentation on offset tracking.

Upvotes: 1

Related Questions