CODEWITHSUNDEEP
hadoopapache-kafka
King David
King David

Reputation: 540

what is the procedure when kafka disk crashed + how to remove the old topics after disk crashed

we have 3 kafka machine in the HDP cluster

kafka01
kafka02
kafka03

kafka version -     0.10.0.2.6

each kafka machine ( kafka03 ) have disk with 18T

and

default.replication.factor=3

the last kafka machines loose the disk ( disk isn't healthy , and we need to replace it )

so we replaced the disk , and we create again the file-system on it and create again the topics

but - as all know before removing the kafka data from the topics , we need to delete all topics in the /var/kafka/kafka-logs

and then it is safe to replace the disk

but this isn't the case because suddenly disk creased and we not deleted the topics

for now kafka broker service stooped after some time , and we think it is because the topics that are not removed before disk replacing

any suggesting regarding this case ?

example how to delete the topic - PlR_TGE_FDS ( that we not did ) , and another 23 topics that not deleted

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper zookeper_server01:2181 --delete --topic PlR_TGE_FDS

Upvotes: 0

Views: 1258

Answers (1)

Elad Leev
Elad Leev

Reputation: 938

Are you using single disk for each broker in your cluster?

In case of crashed disk, the Kafka broker should detect it and shutdown. After you replace the bad disk, just start your broker again, and it should replicate all the data from the other brokers and then took the leadership back to him.

I didn't understand why would you delete the topic in case of disk disk failure, what's the point of replication factor? am I'm missing something?

[1] How to handle disk failures in Kafka? - Hortonworks
[2] Disks and Filesystem - Kafka Documentation

Upvotes: 1

Related Questions