CODEWITHSUNDEEP
apache-kafka
Nadeem
Nadeem

Reputation: 324

How can I check how much disk space is being used by Kafka

My disk space is running out, after syncing 2 millions message in one of my Kafka topic. I even deleted the topic but still the space is somehow seems consumed. Any help of link would be appreciated.

Upvotes: 12

Views: 17503

Answers (4)

Seweryn Habdank-Wojewódzki
Seweryn Habdank-Wojewódzki

Reputation: 2167

  • To delete messages from the topic you shall first set retention time to e.g. 1000ms. This will purge all messages in 1 sec. (see the answer: The correct config key is retention.ms)
  • Delete the topic
  • General think through the retention time for all topics, as default is 7 days, what might be quite much, when you operate with high frequecies.

Upvotes: 1

Andrei
Andrei

Reputation: 520

Another way to determine the size of the topic to send metrics, such as graphite. Then the size can be found as follows: kafka.log.Log.<topic_name>-<partition_number>-Size.value

Upvotes: 1

YaRiK
YaRiK

Reputation: 708

From server.properties

#A comma separated list of directories under which to store log files

log.dirs=/tmp/kafka-logs

du will work but make sure to check all log directories. In fact, any normal unix command that you like to check disk space will work.

Deleting topic does not guarantee deletion. It will put it in "to be deleted" state. If you have any producer/consumer still attached to the topic, it will not be removed. Try the following:

  1. Make sure all brokers that have topic replicas are running
  2. Make sure you allow topic deletion delete.topic.enable = true
  3. Check if topic was deleted bin/kafka-topics.sh --list --zookeeper zookeeper_host:2181

Upvotes: 0

Kamal Chandraprakash
Kamal Chandraprakash

Reputation: 2002

Use the below command to know how much disk space consumed by each topic

du -sh /tmp/kafka-logs/*

Upvotes: 6

Related Questions