CODEWITHSUNDEEP
apache-kafka
George
George

Reputation: 3030

Kafka Sending Message Using Key to decide partition vs manually

Let's say for example I have this Employee POJO:

public class Employee {
  private int id;
  private String name;
}

And I want to send this to a Kafka topic with multiple partitions. So I can use the id field as the key to select the right partition. I can create a ProducerRecord like the following:

ProducerRecord<Integer, Employee> record = new ProducerRecord<>("topic", employee.getId(), employee);

This will send a message to Kafka with the key as a header field and the employee as the payload.

My question is if the key is already included in the payload, and it is a large portion of the payload itself, would it be better to calculate the partition number manually and send it like that?

So it would look something like this:

int partition = defualtPartitioner.partition(...);
ProducerRecord<Integer, Employee> record = new ProducerRecord<>("topic", partition, null, employee);

And in this way would omit the key from the message?

Upvotes: 2

Views: 901

Answers (1)

Gwen Shapira
Gwen Shapira

Reputation: 5158

You could do that, but there are limitations:

  • You can't compact the topic
  • Some streams operations require keys (like joins)
  • Consumers will need to de-serialize the entire message to get the key, so you can't use key for efficient routing or skipping.

As long as you know the trade-offs, and you think that message size really matters... go ahead and omit the key.

Upvotes: 4

Related Questions