CODEWITHSUNDEEP
apache-kafka
Jin Lee
Jin Lee

Reputation: 3512

How does __consumer_offsets keep offset for two consumers?

enter image description here

Just like the above image, if Consumer A consumes from a Partition and Consumer B from a Consumer Group consumes from the same Partition, how does Kafka manage offset in __consumer_offsets?

I want to know how Kafka writes consumer offsets in .index, .log, .timeindex files.

Upvotes: 0

Views: 620

Answers (2)

Jin Lee
Jin Lee

Reputation: 3512

If there are more than one consumer consuming the topic (consumer in a consumer group), Kafka keeps the offset like the below. (I just like images, they help me understand better :) )

enter image description here

enter image description here

enter image description here

In __consumer_offsets-, there are files like below. That's where the offset are recorded.

enter image description here

Index file and Log file contain the following (an example) : Timeindex file is used like index file helping Kafka quickly find messages on the disk.

enter image description here

Lastly, the below diagram can give you an idea how the offset is stored in the log file.

enter image description here

All images are from Google.

Upvotes: 0

Spasoje Petronijević
Spasoje Petronijević

Reputation: 1598

Quote from docs that can be found here: https://kafka.apache.org/documentation/#impl_offsettracking

Kafka provides the option to store all the offsets for a given consumer group in a designated broker.

So, it is per consumer group, not per consumer. Also, this article can be helpful for you: https://medium.com/@felipedutratine/kafka-consumer-offsets-topic-3d5483cda4a6 If you read from consumer offsets topic you will receive data in format [groupId,topicName,partitionNumber] .

Upvotes: 1

Related Questions