CODEWITHSUNDEEP
apache-kafka
Joe.wang
Joe.wang

Reputation: 11793

What does semantic partition function means in Kafka?

All. Forgive me I was just learning the Apache Kafka. When I was reading the document of Kafka. It mentioned a phrase named semantic partition function.

As the document says.

Producers publish data to the topics of their choice. The producer is responsible for choosing which record to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the record). More on the use of partitioning in a second!

What does it mean semantic partition in Kafka? So far I didn't found any more about it in the document. Could someone please help to explain more about it for better understanding? Thanks.

Upvotes: 1

Views: 1722

Answers (1)

ppatierno
ppatierno

Reputation: 10065

When the producer doesn't specify a key for messages, the round robin fashion is used. When a key is specified, the DefaultPartitioner just process an hash on the key (module the number of partitions). If you want, you can use your own partitioner class. The documentation wants just to say that : that the "semantic" for defining the destination partition is up to you, you can develop the "function" (really a partitioner class). For example, instead of using the Kafka key in the message you could have a payload, let me say a JSON, with some data and you want to use one of this info for processing the right destination partition.

Upvotes: 4

Related Questions