CODEWITHSUNDEEP
apache-kafka
gowthamjs23
gowthamjs23

Reputation: 352

programmatically find kafka topic size

I am new to Kafka. I have created a producer using java and sending some messages to a topic. Is there a way to programmatically find the number of messages in a particular topic.

Please share your thoughts.

Upvotes: 2

Views: 1886

Answers (2)

ppatierno
ppatierno

Reputation: 10065

What vahid sayd could be a solution but you can also use the following one :

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic --time -1

then you have to sum the count values.

In any case please consider that my and vahid solutions don't work if log compaction is active because in this case there could be some "gaps" between offsets so calculating just the difference doesn't provide you the number of messages.

Upvotes: 1

vahid
vahid

Reputation: 1218

KafkaConsumer API provides interfaces that can help you. There is beginningOffsets(Collection<TopicPartition> partitions) and endOffsets(Collection<TopicPartition> partitions). You can call these methods for all partitions in your topic, and then find the difference between end offset and beginning offset for each topic partition. Then aggregate across all topic partitions. That should give you the number of messages in the topic. (reference)

Upvotes: 2

Related Questions