CODEWITHSUNDEEP
apache-kafkaapache-zookeeper
malachi54
malachi54

Reputation: 77

Can a Kafka-Cluster be cut in half?

Scenario: you have a Kafka-Cluster in different DCs but they are configured as one cluster. So there is no mirroring through MirrorMaker or something liket hat. The DCs are not very far from eatch other. But they are physically separated.

Now what do you have to do to ensure that the cluster is failsafe on BOTH SIDES if the connection between those two DCs is down? So on BOTH sides the producers and consumer should still work.

I would guess: you need multiple Zookeepers on both sides and multiple Kafka-Nodes.

But is that enough? Does the cluster heal itself after getting reconnected?

Thanks in advance.

Upvotes: 0

Views: 609

Answers (1)

Mickael Maison
Mickael Maison

Reputation: 26895

I'm assuming your datacenters that "are not very far from eatch other" are basically Availability Zones (AZs).

It's pretty common to spread a cluster over multiples AZs. However it's usually not desired or possible that each "slice" can live on its own.

The immediate issue is Zookeeper which by design prevents split-brain scenarios. So if a ZK cluster is split only one "slice" (at best) will carry on working. So the brokers that are on a side of the non working ZK clusters will be non functional.

Then let's say it was possible to have both sides keep working. What happens when you joins both sides again?

Data is likely to have diverged as clients wrote data to each side separately. Now you could have the same partition with different messages for the same offset and no way to resolve the conflict as both options are "valid".

I hope this shows why this is not a possible solution. In practice, if an AZ goes offline, it is non functional until it is brought back online.

Clients that were connected to the offline AZ should reconnect to the other AZ (using multiple bootstrap servers) and clients that were in the failed AZ should be reprovisioned into another one.

If configured correctly, Kafka can survive an AZ outage (even though in practice, it's best to have 3 AZs) and keep all resources available. Also in this scenario, the cluster will automatically return to a good state when the failed AZ returns.

Upvotes: 2

Related Questions