How many partitions should I create for kafka consumers?

Question

Currently I am having only one partition of topic and only one consumer. But my consumer runs slow so I want to add new consumers to the same topic.

But at the same time I also want all the consumers to process exclusive set of messages.

I am not able to determine how many partitions should I create so that I can add consumers whenever I want to increase throughput and to process exclusive messages.

Answer 1

If you have N partitions for a topic, you can start a consumer group with N consumers, and they will each get to work on one of the partitions.

Each consumer will get a disparate set of messages (unless there is fail-over involved, then some messages may be delivered more than once) that way, and if the workload is evenly spread amongst all partitions, this will load-balance properly.

If you have fewer than N consumers, some of them will get more than one partition, but they will still not share partitions and each will get roughly the same number of partitions.

If you have more than N consumers, some will stay idle (but can act as hot standbys in case one of the others drops out).

The key to all this is that you put these consumers into the same consumer group. That is the way to indicate to Kafka that they should divide the partitions between them. If you started N consumer groups instead, each group would get all messages in parallel (each message would be processed N times).

The main downside to partitioning is that you lose the total ordering of messages. Messages are delivered in-order only with respect to other messages in the same partition. This may or may not be a problem for your application (and if it is, you can re-arrange things in your application by looking at timestamps).