CODEWITHSUNDEEP
apache-kafkascalability
vartec
vartec

Reputation: 134551

Does Kafka scale well for big number of clients?

I know Kafka can handle tons of traffic. However, how well does it scale for big number of concurrent clients?

Each client would have their own unique group_id (and as consequence Kafka would be keeping track of each one's offsets).

Would that be an issue for Kafka 0.9+ with offsets stored internally?

Would that be an issue for Kafka 0.8 with offsets stored in Zookeeper?

Upvotes: 2

Views: 2344

Answers (1)

miguno
miguno

Reputation: 15057

Some Kafka users such as LinkedIn have reported in the past that a single Kafka broker can support ~10K client connections. This number may vary depending on hardware, configuration, etc.

As long as the request rate is not too high, the limiting factor is probably just the open-file-descriptors limit as configured in the operating system, see e.g. http://docs.confluent.io/current/kafka/deployment.html#file-descriptors-and-mmap for more information.

Upvotes: 2

Related Questions