CODEWITHSUNDEEP
javaapache-kafka
john
john

Reputation: 11669

writing our own custom partitioner for kafka

I need to write my own partitioner basis on the key we have. It looks like we can write our own custom partitioner.

From the Kafka main site they say

Producers publish data to the topics of their choice. The producer is responsible for choosing which record to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the record). More on the use of partitioning in a second!

In my case, given a topic we will have 10 partitions for it so we want to use this formula to decide which partition the data should go into.

partition = client_id % MOD 10

Here client_id will be the key and it will always be numerical value, it will be long data type always. How can I write our own custom partitioner for kafka producer which can tell me what partition I should use given for client_id.

I saw that we have to implement Partitioner class and make some changes in the partition method but I am not sure how to use keyBytes variable to figure out the partition should it go into basis on above formula.

@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes,
  Cluster cluster) {
    // TODO Auto-generated method stub
    return 0;
}

I am running Kafka 0.10.0.0 version.

Upvotes: 4

Views: 3918

Answers (2)

jimijazz
jimijazz

Reputation: 2257

If you specify the key (i.e. it is not null) but not the partitioner, Kafka will do exactly what you intend to do.

Here is an excerpt taken from the book "Kafka: The definitive Guide" by O'Reilly Media:

If a key exists and the default partitioner is used, Kafka will hash the key (using its own hash algorithm, so hash values will not change when Java is upgraded), and use the result to map the message to a specific partition.

Upvotes: 0

amethystic
amethystic

Reputation: 7089

keyBytes is the serialized key to partition on. You could use 'key'（with Object type) directly to do the partition.

Upvotes: 3

Related Questions