CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
JackJack
JackJack

Reputation: 75

Kafka - Standalone sever - How to decide partitions?

I have a standalone Kafka setup with single disk. planning to stream over million records. How to decide partitions for my topic for better through-put? has to be 1 partition?

Is it recommended to have multiple partitions for a topic on standalone Kafka server?

Upvotes: 0

Views: 449

Answers (2)

Javier Yáñez
Javier Yáñez

Reputation: 631

There are two main factors to consider:

Number of producers and consumers
Each client, producer or consumer, can only connect to one partition. For this reason, the number of partitions must be at least the max(number of producers, number of consumers).

Throughput
You must determine the troughput to calculate how many consumers should be in the consumer group. The combined reading capacity of consumers should be at least as high as the combined writing capacity of producers.

Upvotes: 1

Mike
Mike

Reputation: 531

Yes you need multiple partitions even for a single node kafka cluster. That is because you can only have as many consumers as you have partitions. If you have a single partition then you can only have a single consumer, and that will limit throughput. Especially if you want to stream millions of rows (although the period for those is not specified). The only real downside to this is that messages are only consumed in order within the same partition. Other than that, you should go with multiple partitions. You will need to estimate the throughput of a single consumer in order to calculate the partitions, then maybe add one or 2 on top of that. You can still add partitions later but it's probably better to try to start with the right amount first and change later as you learn more or as your volume increases/decreases.

Upvotes: 1

Related Questions