CODEWITHSUNDEEP
javaapache-kafkakafka-producer-api
mechanikos
mechanikos

Reputation: 777

Kafka addtional partition key or partitoner without key

I`m new in Kafka and have some problems. I know that we can implenent Partitioner class with own partion logic which return specific partiton depending on message key. Also we can set log compaction policy under which only the latest version of messages with same key are stored. But I need to have different keys for message for this actions. For example, we have entity with id and address(city_id). I want to choose partition depending on city_id and store only latest info about people with same id. Is there any way to solve this problem? Sorry for my bad English, I really want to learn Kafka.

Upvotes: 0

Views: 400

Answers (1)

Marko Bonaci
Marko Bonaci

Reputation: 5708

Is it absolutely necessary that all messages with the same city go to the same partition? If yes, why? Can you present your use case that requires that?

If not, you could just include both city_id and person_id in hash calculation. That way compaction should remove all but the latest person message (provided that you have 1 to 1 mapping from person to city, which is usually true in schemas)?

Upvotes: 1

Related Questions