CODEWITHSUNDEEP
apache-kafka
Apolo-11
Apolo-11

Reputation: 99

Kafka - How to recover if a partition is lost?

I have 4 Kafka nodes in a cluster, one topic split to 40 partitions and replica count 2. Kafka version is 2.3.1.

How can I recover from the situation when two Kafka nodes die at the same time, it is not possible to start them again and Kafka logs are lost?

I'm sure that I lose some data because some partitions are lost (some partitions have replicas only on the died nodes).

I tried to add two new Kafka nodes and reassign partitions to all 4 available Kafka nodes. Finally, lost partitions are not reassigned to the two new Kafka nodes. Clients cannot publish data that go to lost partitions.

Upvotes: 3

Views: 2953

Answers (1)

HISI
HISI

Reputation: 4797

Kafka recovers by himself the losing partitions only if those partitions still have at least one alive replica that was previously in sync. Otherwise unclean.leader.election must be enabled on the brokers to move the leader to an out of sync replica

Since partitions had only 2 replica and you lost 2 nodes, you might lose some partitions.

You can replace 2 replica by 4 replica to more reliability

The two added nodes should have the same id as the previous ones to be able to pull replica.

Upvotes: 5

Related Questions