CODEWITHSUNDEEP
apache-kafkapartition
n4feng
n4feng

Reputation: 2454

kafka message re-balance when broker shutdown

A general question. Assume a topic has 3 kafka partitions on different servers (brokers), each partition has 10 message with offset as its timestamp (0,1,...,9, greater number means stayed shorter time in partition, also means is newly came message). Let's say one partition happen to shut down since the server is done. What's the strategy for Kafka to re-balance the 10 message in the shut down partition into other partitions?

Visually, we have

broker 1 partition: |1-0|1-1|1-2|1-3|1-4|1-5|1-6|1-7|1-8|1-9|

broker 2 partition: |2-0|2-1|2-2|2-3|2-4|2-5|2-6|2-7|2-8|2-9|

broker 3 partition: |3-0|3-1|3-2|3-3|3-4|3-5|3-6|3-7|3-8|3-9|

Now if broker 3 is done, how will 3-0 to 3-9 be inserted into broker 1 and broker 2?

( My assumption is by default it will be spread half half randomly and inserted based on timestamp of broker 3, attached to tail of broker 1 and 2, and maybe there is somewhere one can configure behavior by code?)

Thanks in advance.

Upvotes: 1

Views: 1353

Answers (2)

Jin Lee
Jin Lee

Reputation: 3512

The below diagram will help you understand how Kafka replicates partitions. If one broker is down, the consumer can read from the other broker because Kafka has a replication ability. (Of course, you need to set it like below) For example, if broker 1 dies, broker 2 will become a leader of topic1-part1, and a consumer can read from it.

enter image description here

Zookeeper will know if a broker( partition) is down, it will appoint another leader.

Upvotes: 2

Mickael Maison
Mickael Maison

Reputation: 26885

If a partition only exists on a single broker (replication factor 1) then when this broker is offline, the partition is not available. This is what you drew in your question.

To keep data available even when brokers go down you have to create topics with a replication factor greater than 1.

Then the data of the partition will be replicated onto several brokers and if one of them go offline, user traffic will be rediected to the available replicas.

I suggest you to go through the Replication section in the docs to understand how this works.

Upvotes: 5

Related Questions