CODEWITHSUNDEEP
apache-kafka
ecwdw 23e3e23e
ecwdw 23e3e23e

Reputation: 475

How does Kafka choose a partition for key?

Suppose the number of partitions is fixed for a topic. And suppose I provide a key --- an array of bytes --- when publishing a message. Kafka will map the key onto a topic partition 0,1,2,3,...N-1.

I'd like to make a function that does the same thing: given N, and the key return the same partition number as would Kafka.

Is this this just (in pseudo-code) murmur3(key) % N?

TIA

Upvotes: 3

Views: 6435

Answers (2)

Bartosz Wardziński
Bartosz Wardziński

Reputation: 6583

DefaultPartitioner uses murmur3 function to calculate partition for messages with not null keys.

You can just extract algorithm and added to your code:

import org.apache.kafka.common.utils.Utils;
Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

Second option is to just use DefaultPartitioner:

DefaultPartitioner partitioner = new DefaultPartitioner();
int partition = partitioner.partition(???, ???, ???, ???, ???, ???);

Upvotes: 4

Jin Lee
Jin Lee

Reputation: 3512

As far as I know, Kafka hashes a key and partition according to that hash value. Here is a helpful diagram and an example on how.

Utils.abs(java.util.Arrays.hashCode(key.asInstanceOf[Array[Byte])) % numPartitions

enter image description here

Just found another good example of partitioning!

enter image description here

Upvotes: 3

Related Questions