CODEWITHSUNDEEP
apache-kafka
supernatural
supernatural

Reputation: 1187

What is a kafka broker physically?

A kafka cluster is composed of multiple brokers.
Q1. How does a single kafka broker look like physically, is it a machine/pc/node ?
Q2. If it is a node, then why it is named explicitly as kafka broker as kafka cluster should be enough right? Say a kafka cluster, that's it?
Q3. Why is there a concept of kafka broker, why don't we have a single kafka cluster and if we want to scale then why not just add more nodes on it or more hardware on that cluster?

Upvotes: 2

Views: 2933

Answers (1)

JavaTechnical
JavaTechnical

Reputation: 9347

  1. Kafka broker is a logical process, not a phsyical entity. It can be run in a physical machine or a virtual machine, or in a container. Though some people, might use the term broker and node interchangeably. But, as far as the definition goes, a node is a typically a physical entity, a machine or a VM, but a broker is a process. We can run multiple brokers in one physical node also.
  2. A node represents one entity in a cluster. A Kafka cluster comprises of several Kafka brokers (from 1 to many). If it is a single broker cluster, then we can still call it a Kafka cluster or simply Kafka broker. Though, sometimes when people say a Kafka cluster, they mean every thing that is needed to run Kafka i.e. including the Zookeeper cluster also.
  3. Adding more nodes is a way of scaling, but it isn't enough if we just add physical nodes, we need to run our Kafka broker processes in it and only then we can distribute the load among Kafka brokers. People use only a single Kafka cluster that is comprised of many brokers. Some organizations may also maintain multiple Kafka clusters, most typically for redundancy.

Upvotes: 7

Related Questions