CODEWITHSUNDEEP
apache-kafkarecovery
HansHarhoff
HansHarhoff

Reputation: 1997

What happens if you delete Kafka snapshot files?

The question is simple. What will happen if you delete the Kafka snapshot files in the Kafka log dir. Will Kafka be able to start? Will it have to do a slow rebuild of something?

Bonus question what exactly do the snapshot files contain?

Background for this question

I have a cluster that has been down for a couple of days due to simultaneous downtime on all brokers and a resulting corrupted broker. Now when it starts it is silent for hours (in the log file no new messages). By inspecting the JVM I have found that all the (very limited) cpu usage is spent in loadproducersfromlog function/method. By reading the comments above it is suggested that this is an attempt to recover producer state from the snapshots. I do not care about this. I just want my broker back so I am thinking if I can simply delete the snapshots to get Kafka started again.

Upvotes: 4

Views: 3729

Answers (2)

Prasanth Nair
Prasanth Nair

Reputation: 509

If snapshot files are deleted, during start up method log.loadSegmentFiles(), all messages in the partition will have to be read to recreate the snapshot even if log and index files are present. This will increase the time to load partition.

For contents of snapshot file, please refer writeSnapshot() in ProducerStateManager. https://github.com/apache/kafka/blob/980b725bb09ee42469534bf50d01118ce650880a/core/src/main/scala/kafka/log/ProducerStateManager.scala

Upvotes: 4

Tiago Peres
Tiago Peres

Reputation: 15476

Parameter log.dir defines where topics (ie, data) is stored (supplemental for log.dirs property).

A snapshot basically gives you a copy of your data at one point in time.

In a situation like yours, instead of waiting for a response, you could:

  • change the log.dirs path, restart everything and see how it goes;

  • backup the snapshots, saving them in a different location, delete them all from the previous one and see how it goes.

After that you're meant to be able to start up Kafka.

Upvotes: 0

Related Questions