CODEWITHSUNDEEP
apache-kafkapartitioningkafka-producer-api
Shankar
Shankar

Reputation: 8957

Kafka - Message with different key stored in same partition

I am trying to store Messages with different key to different partition.

For example:

ProducerRecord<String, String> rec1 = new ProducerRecord<String, String>("topic", "key1", line);
ProducerRecord<String, String> rec2 = new ProducerRecord<String, String>("topic", "key2", line);
producer.send(rec1);
producer.send(rec2);

But when i try to run my Producer class, it always stored in single partition.

As per documentation, DefaultPartitioner uses message key hash code to find the partition. I also saw this question Kafka partition key not working properly‏, but i cannot find ByteArrayPartitioner class in 0.9.x version of Kafka Client library.

props.put("partitioner.class", "kafka.producer.ByteArrayPartitioner")

Update: I am creating the topic on the fly using code.

If i create a topic with partitions manually, then its working fine.

Upvotes: 1

Views: 2813

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

If topics are created "on the fly", the are created with number of partitions according to num.partitions parameters (with default value 1). And if you have only one partitions, all data will go to this single partitions.

However, keep in mind, even if you have multiple partitions, a partitions can still get different keys assigned! Even if you have num-partitions == num-distinct-keys there might be hash collisions, assigning two different keys to the same partitions (and leaving some partitions empty).

If you want to ensure that different keys always go to different partitions, you need to use a consumer partitioner or specify the partition number directly.

Upvotes: 3

Related Questions