CODEWITHSUNDEEP
apache-kafka
Louis F.
Louis F.

Reputation: 2048

How do messages get consumed in Kafka when there are more partitions than consumers?

This is a very basic question about Kafka and how messages are consumed but I was not able to find any answer on this point, unfortunately :'(.

Let's say I want to over partition a bit, so I would get 10 times more partitions than consumers. Over partition is required as I want to be able to scale (process more messages in parallel in the future).

1 topic divided in 1000 partitions consumed by 100 consumers =-> each consumer get assigned 10 partitions.

My questions are :

I was not able to find any answer on this point. And sorry if it's a duplicate !

Upvotes: 2

Views: 295

Answers (1)

Nautilus
Nautilus

Reputation: 2286

  • The consumption of messages is done in a round robin way between the set of partitions assigned to a Consumer.

  • No, there is no guarantee that the messages are going to be consumed evenly between partitions. In the documentation for KIP-41 you can see an explanation on how this will work when you can specify the number of messages

Upvotes: 1

Related Questions