CODEWITHSUNDEEP
apache-kafka
Judy
Judy

Reputation: 1869

kafka machines in the cluster and kafka communications

We have kafka cluster with 3 kafka brokers nodes and 3 zookeepers servers

kafka version - 10.1 ( hortonworks )

from my understanding since all meta data is located on the zookeeper servers , and kafka brokers are using this data ( kafka talk with zookeeper server via port 2181 )

I just wondering if each kafka machine talk with other kafka in the cluster , or maybe kafka are get/put the data only on/from the zookeepers servers ?

So dose kafka service need to communicate with other kafka in the cluster ? , Or maybe kafka machines get all is need only from the zookeepers server ?

enter image description here

Upvotes: 1

Views: 574

Answers (1)

Nether
Nether

Reputation: 1160

Kafka brokers certainly need to communicate with each other, most importantly to replica data. Data produced to Kafka is replicated across brokers for fault-tolerance and data durability. Partition followers send FetchRequests to partition leaders in order to replicate the data.

Additionally, the Controller broker sends a LeaderAndIsr request to brokers whenever a partition leader/follower is changed - that's how it informs brokers to start leading a partition or replicating it.

I would recommend these two introductory articles of mine in order to help you get more context:

https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1

https://hackernoon.com/apache-kafkas-distributed-system-firefighter-the-controller-broker-1afca1eae302

Upvotes: 2

Related Questions