CODEWITHSUNDEEP
apache-kafka
Rahul Gupta
Rahul Gupta

Reputation: 1802

Under what circumstance will kafka overflow?

I am testing a POC where data will be sent from 50+ servers from UDP port to kafka client (Will use UDP-kafka bridge). Each servers generates ~2000 messages/sec and that adds up to 100K+ messages/sec for all 50+ servers. My question here is

  1. What will happen if kafka is not able to ingest all those messages? Will that create a backlog on my 50+ servers?
  2. If I have a kafka cluster, how do I decide which server sends message to which broker?

Upvotes: 1

Views: 1088

Answers (1)

Simon Clark
Simon Clark

Reputation: 654

1) It depends what you mean by "not able" to ingest. If you mean it's bottlenecked by the network, then you'll likely just get timeouts when trying to produce some of your messages. If you have retries (documented here) set up to anything other than the default (0) then it will attempt to send the message that many times. You can also customise the request.timeout.ms, the default value of this being 30 seconds. If you mean in terms of disk space, the broker will probably crash.

2) You do this in a roundabout way. When you create a topic with N partitions, the brokers will create those partitions on whichever server has the least usage at that time, one by one. This means that, if you have 10 brokers, and you create a 10 partition topic, each broker should receive one partition. Now, you can specify which partition to send a message to, but you can't specify which broker that partition will be on. It's possible to find out which broker a partition is on after you have created it, so you could create a map of partitions to brokers and use that to choose which broker to send to.

Upvotes: 2

Related Questions