CODEWITHSUNDEEP
apache-kafkakafka-rest
Demid Demidov
Demid Demidov

Reputation: 51

Kafka REST Proxy API uneven distribution of messages

I have two partitions and messages in avro-format. I send the messages via the Kafka REST proxy API. I use the key for the messages. The key is a string. For example, there are my keys:

41:46-300, 41:45-300, 41:44-300, 41:43-300, 41:42-300.

But the messages are uneven distributed. In the partition 0 there are messages with keys 41:46-300, 41:45-300, 41:44-300, and 41:43-300. And in the partition 1, there are only messages with the key 41:42-300.

Kafka version: 2.4

Why does this happen?

Upvotes: 0

Views: 80

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 192013

Kafka uses Murmur2 hashing to distribute keys, not an evenly distributed round-robin mechanism.

So, this means all events in the same partition ended up with hashes that modulo'd into that partition.

Upvotes: 0

Related Questions