CODEWITHSUNDEEP
apache-kafkacloudera-manager
unreal
unreal

Reputation: 53

Large number of messages from Kafka consumer_offset topic

I have been playing around with Kafka for a few months now and I have noticed that I have a rather large number of message/sec under the internal Kafka topic called __consumer_offsets. Now we have a hand full of other topics that are rather small in terms of the number of message across the brokers. At the most we can see ~1k message/sec on a couple topics, but I almost always see 15-20k message/sec from _consumer_offsets. Can someone point me to something that will explain this or provide some insight in why the consumer_offsets topic has so many messages.

enter image description here

Thanks

Upvotes: 3

Views: 1215

Answers (2)

Cloud66
Cloud66

Reputation: 51

I have The same. Number of messages in all other topic is insignificant in comparison to __consumer_offsets topic. Is it problem because high number of commits by producers ? If not, we should hide this topic from monitoring graph for clear seeing another topics.

Upvotes: 0

Robin Moffatt
Robin Moffatt

Reputation: 32110

This is how Apache Kafka works. See http://kafka.apache.org/documentation.html#impl_offsettracking for more details.

Upvotes: 1

Related Questions