CODEWITHSUNDEEP
apache-kafka
user830818
user830818

Reputation: 417

Kafka Inter Broker Communication

I understand producer/consumers need to talk to brokers to know leader for partition. Brokers talk to zk to tell they joined the cluster.

Is it true that

  1. Brokers know who is the leader for a given partition from zk
  2. zk detects broker left/died. Then it re-elects leader and sends new leader info to all brokers

Question:

  1. why do we need brokers to communicate with each other? Is it just so tehy can move partitions around or do they also query metadata from each other. If so what would be example of metadata exchange

Upvotes: 2

Views: 5594

Answers (2)

Milos Pajic
Milos Pajic

Reputation: 336

Producers/ consumers request metadata from one of the brokers ( as each one of them caches it) and that is how they know who is the leader for a partition.

Regarding "is it true that" section:

  1. Brokers know who is the leader for the given partition thanks to zk and one of them. To be more precise, one of them decides who will be a leader. That broker is called controller. The first broker that connects to zookeeper becomes a controller and his role is to decide which broker will be a leader and which ones will be replicas and to inform them about it. Controller itself is not excluded from this process. It is a broker like any other with this special responsibilities of choosing leaders and replicas
  2. zk indeed detects when a broker dies/ leaves but it doesn't reelect leader. It is controller responsibility. When one of the brokers leaves a cluster, controller gets information from zk and it starts reassignment

About your question - brokers do communicate with each other ( replicas are reading the messages from leaders, controller is informing other brokers about changes), but they do not exchange metadata among themselves - they write metadata to a zookeeper

Upvotes: 4

sandeep rawat
sandeep rawat

Reputation: 4957

A Broker is a Kafka server that runs in a Kafka Cluster

"A Kafka cluster is made up of multiple Kafka Brokers. Each Kafka Broker has a unique ID (number). Kafka Brokers contain topic log partitions. Connecting to one broker bootstraps a client to the entire Kafka cluster"

Each broker holds a number of partitions and each of these partitions can be either a leader or a replica for a topic. All writes and reads to a topic go through the leader and the leader coordinates updating replicas with new data. If a leader fails, a replica takes over as the new leader.

Upvotes: 1

Related Questions