Kafka rack awareness feature

Question

Is kafka rack awareness activated by default on my cluster? If I configure some brokers with the broker.rack and let other brokers without broker.rack, does kafka rack awareness affect my cluster?

Answer 1

I tried this scenario(where some brokers have broker.rack property set and others don't have). Topic creation is failing with below error:

Error: Not all brokers have rack information for replica rack aware assignment. - Unknown broker error (unknown)

Answer 2

Here is the answer I get after some research:

When you configure some brokers with broker.rack and let others without broker.rack, Manual Topic creation will fail and generate an error (Automatic topic creation ignore the rack information).

To fix this issue:

Option 1: place the property broker.rack=null on all brokers.

Option 2: edit the property broker.rack on all brokers where is missing, will the id of the rack.

Option 3: add the property --disable-rack-aware to your kafka-topics command.

For more details about rack awareness, Here is the kafka confluent documentation

Answer 3

As for your question, it won't affect your Kafka cluster.

Each broker has its own configuration file. So, if u are adding the broker.rack configuration on one broker and not on other it won't cause any harm to the cluster. But for best practices, if you want to have a fault-tolerant system and that your data should not be lost from other leader partition on different broker then you have to apply the configuration on all the brokers.

For more knowledge on rack awareness and replication go through the Kafka docs: https://kafka.apache.org/documentation/#basic_ops_racks

Answer 4

Default value of broker.rack is null. So if you define broker.rack for some brokers Kafka will have two options for replication assignment (your_rack_id and null). So I guess there is no reason to NOT consider these two options during assignment.