CODEWITHSUNDEEP
apache-kafka
sanjeev
sanjeev

Reputation: 107

Kafka producer message flow

I have a topic called Topic1 with two partitions. Assume that serverA is leader for Topic1, partition1 and serverB is a follower.

What will happen if in my client I publish to serverB (in the broker.list I specify only serverB)? How does the message propagate? Is it sent to serverB and then to serverA.

Upvotes: 2

Views: 1926

Answers (2)

Vishal John
Vishal John

Reputation: 4382

To publish a message to a partition, the client first finds the leader of the partition from Zookeeper and sends the message to the leader. The leader writes the message to its local log. Each follower constantly pulls new messages from the leader using a single socket channel.The follower writes each received message to its own log and sends an acknowledgment back to the leader. Once the leader receives the acknowledgment from all replicas in ISR, the message is committed

So to answer your question, if the client publishes to serverB, it consults zookeeper to know the leader for Topic1 and partition1. Zookeeper responds saying serverA is the leader for partition1. So the client sends the message to serverA.(Here I assumed that the partitioner will send the message to partition1.)

All these are handled by the kafka producer. End user application need not worry about these details.

You can read more about this here

Upvotes: 1

Chris Gerken
Chris Gerken

Reputation: 16392

I've found this document to be very helpful in explaining what's going on internally with Kafka.

There's an API used by the producer to ask any one Kafka server for the list of partitions and all metadata for those partitions. That metadata includes the leader broker for each partition. The producer calls the partitioner to get the target partition and then talks directly with that partition's Kafka broker leader to write the message onto the partition. The leader will handle communications with any other brokers managing replicas of the partition.

Upvotes: 1

Related Questions