CODEWITHSUNDEEP
apache-kafka
Pritam Kadam
Pritam Kadam

Reputation: 2527

How many bootstrap servers to provide for large Kafka cluster

I have a use case where my Kafka cluster will have 1000 brokers and I am writing Kafka client. In order to write client, i need to provide brokers list.

Question is, what are the recommended guidelines to provide brokers list in client?

Is there any proxy like service available in kafka which we can give to client? - that proxy will know all the brokers in cluster and connect client to appropriate broker. - like in redis world, we have twemproxy (nutcracker) - confluent-rest-api can act as proxy?

Is it recommended to provide any specific number of brokers in client, for example provide list of 3 brokers even though cluster has 1000 nodes? - what if provided brokers gets crashed? - what if provided brokers restarts and there location/ip changes?

Upvotes: 0

Views: 381

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

The list of broker URL you pass to the client are only to bootstrap the client. Thus, the client will automatically learn about all other available brokers automatically, and also connect to the correct brokers it need to "talk to".

Thus, if the client is already running, the those brokers go down, the client will not even notice. Only if all those brokers are down at the same time, and you startup the client, the client will "hang" as it cannot connect to the cluster and eventually time out.

It's recommended to provide at least 3 broker URLs to "survive" the outage of 2 brokers. But you can also provide more if you need a higher level of resilience.

Upvotes: 1

Related Questions