CODEWITHSUNDEEP
apache-kafka
John
John

Reputation: 11831

Kafka rack awareness and locations of ISR

I want to build a HA Kafka cluster, where the cluster needs to span 2 availability zones.

I want to be able to continue to read and write to a topic, even if all the brokers in an AZ go down.

If I have at least 2 brokers in each AZ, a replication factor of 3, min ISR of 2 and acks set to All, then I think that a producer write will be acked when one other broker other than the leader also acks the write. Does the rack aware algorithm enforce that the ISR must be located in the other AZ? The docs just mention replicas, not the ISR.

Will this enable me to continue reading and writing in the event of the loss of an AZ? If not, what is needed to achieve this?

Upvotes: 2

Views: 1960

Answers (1)

Hans Jespersen
Hans Jespersen

Reputation: 8335

If you want a true HA Kafka cluster you need to start with an HA Zookeeper ensemble which typically means 3 Availability Zones because (unlike Kafka brokers) Zookeeper nodes need a quorum (a majority of the original nodes) to operate and you can’t have a majority when half of your nodes are down.

The reason Zookeeper is important is that a proper HA Kafka cluster should not just allow reads and writes after a failure, but also allow new topic creation and new leader elections, both of which require Zookeeper to be operational.

Upvotes: 1

Related Questions