CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
Nilay Narlawar
Nilay Narlawar

Reputation: 35

How client will automatically detect a new leader when the primary one goes down in Kafka?

Consider the below scenario:

  1. I have a Kakfa broker cluster(localhost:9002,localhost:9003,localhost:9004,localhost:9005).
  2. Let's say localhost:9002 is my primary(leader) for the cluster.
  3. Now my producer is producing data and sending it to the broker(localhost:9002).
  4. If my primary broker(localhost:9002) goes down, with the help of Zookeeper or some other consensus algorithm new leader will be elected(consider localhost:9003 is now the new leader).

So, in the above scenario can someone please explain to me how the Kafka client(producer) will get notified about the new broker configuration(localhost:9003) and how it will connect to the new leaders and start producing data again.

Upvotes: 0

Views: 1945

Answers (1)

Michael Heil
Michael Heil

Reputation: 18475

Kafka clients are receiving the necessary meta information from the cluster automatically on each request when reading from or writing to a topic in case of a leadership change.

In general, the client sends a (read/write) request to one of the bootstrap server, listed in the configuration bootstrap.servers. This initial request (hence called bootstrap) returns the details on which broker the topic partition leader is located so that the client can communicate directly with that broker. Each individual broker contains all meta information for the entire cluster, meaning also having the knowledge on the partition leader of other brokers.

Now, if one of your broker goes down and the leadership of a topic partition switches, your producer will get notified about it through that mechanism.

There is a KafkaProducer configuration called metadata.max.age.ms which you can modify to update metadata on your producer even if there is no leadership change happening:

"Controls how long the producer will cache metadata for a topic that's idle. If the elapsed time since a topic was last produced to exceeds the metadata idle duration, then the topic's metadata is forgotten and the next access to it will force a metadata fetch request."

Just a few notes on your question:

  • The term "Kafka broker cluster" does not really exists. You have a Kafka cluster containing one or multiple Kafka brokers.

  • You do not have a broker as a "primary(leader) for the cluster" but you have for each TopicPartition a leader. Maybe you mean the Controller which is located on one of the brokers within your cluster.

Upvotes: 3

Related Questions