CODEWITHSUNDEEP
apache-kafkaconfluent-platform
darshan kamat
darshan kamat

Reputation: 424

Kafka automatic replication to new node

I am a newbie to Kafka and wanted to understand the behaviour of apache Kafka in below scenario.

Consider I have the topic with 3:

As per my understanding if broker 1 goes down there is no harm and no data loss as isr=2 and writes will be successful.

If node 1 comes back up it will again follow the leader and catch up.

My question is if node 1 never comes back up..also it's removed from isr list... My replication factor desired is 3... and if I add new node 4 ..how to automatically make node 4 copy the partitions from failed node 1...so that replication of 3 is still maintained?

Upvotes: 1

Views: 477

Answers (1)

H.Ç.T
H.Ç.T

Reputation: 3579

Topic replicas stored in zookeper and if you add new broker you should update replicas value for topic(s). (As far as I know there is no way to do it automatically)

But you can do it manually by using kafka-reassign-partitions.sh tool.

Steps:

  • Create a json file to represent your desired replicas for partitions. For example:
{"version":1,
  "partitions":[
     {"topic":"YourTopic","partition":0,"replicas":[3,2,4]},
     {"topic":"YourTopic","partition":1,"replicas":[2,4,3]},
     {"topic":"YourTopic","partition":2,"replicas":[4,3,2]}
]}

  • Execute this command to reassign partitions.

    ./kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file my_file.json --execute

Upvotes: 1

Related Questions