CODEWITHSUNDEEP
apache-kafka
Vitalii Kelemen
Vitalii Kelemen

Reputation: 155

How many Producers can I use to write to a single topic

I have a web application which put messages into a Kafka topic. There are a lot of instances of this application (200) and each of them contains it's own Kafka Producer.

Questions:

  1. Does there exist any upper bound of Producers amount per topic?
  2. Does the number of Producers impact on Kafka performance? If yes, how?
  3. What is the best practice for Producers? One synchronous producer per application, an asynchronous producer, or a custom pool of sync producers?

Upvotes: 6

Views: 1176

Answers (1)

nelsonda
nelsonda

Reputation: 1188

Is exists any upper bound of Producers amount per topic?

The only limitation I am aware of is the number of available IP addresses. It is unlikely you'd bump into any practical limit in your described application.

Does Producer amount impact on Kafka performance? If yes, how?

No, all other things being equal (traffic volume, asynchronous vs synchronous (including batch size / time constraints), etc).

Presumably there's some overhead somewhere for the connection, but its small enough that I've never managed to notice it.

What is Producer best practice (One sync producer per application, async producer or custom pool of sync producers)

Depends a whole bunch on your use case, which I am not clear on. For the most part, asynchronous > synchronous. If you choose to use asynchronous, then you have to deal with the risks of batching on the producers (ie data loss), and the delays associated with building up enough messages for a batch / waiting for the batch timeout to trigger. Those delays could be significant if your use case is sufficiently demanding.

Upvotes: 1

Related Questions