CODEWITHSUNDEEP
apache-kafkaapache-kafka-streams
R.B.
R.B.

Reputation: 537

Where does kafka store offsets of internal topics?

__consumer_offsets store offsets of all kafka topics except internal topics such as *-changelog topics in case of streams. Where is this data stored?

Upvotes: 0

Views: 1945

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

The term "internal topic" has two different meanings in Kafka:

  1. Brokers: an internal topic is a topic that the cluster uses (like __consumer_offsets). A client cannot read/write from/to this topic.
  2. Kafka Streams: topics that Kafka Streams creates automatically, are called internal topics, too.

However, those -changelog and -repartition topics that are "internal" topics from a Kafka Streams point of view, are regular topics from a broker point of view. Hence, offsets for both are stored in __consumer_offsets like for any other topic.

Note, that Kafka Streams will only commit offsets for -repartition topics, though. For -changelog topics no offsets are ever committed (Kafka Streams does some offset tracking on the client side though, and writes -changelog offsets into a local .checkpoint file).

Upvotes: 2

Related Questions