CODEWITHSUNDEEP
apache-kafkapartitioning
Kumar
Kumar

Reputation: 1666

Kafka Topic, Partitioning and Topic Log Compaction

I am new to Kafka. I have a question regarding how to partition the topic.

I have Kafka topic and plan to have 2 partitions into it. I have two operations - Project Create / Update and Project Validate.

Project Create / Update operation is long one and takes time to complete. Project Validate is faster operation and is expected to return result quickly.

I am planning to have 2 different keys - One for Create / Update operation and one for Validate operation, so that each operation goes to individual partition and eventually listened by one consumer in consumer group for performance.

I came to know about kafka compaction, where the records are compacted in case if they have same key. This leads to a problem with my initial approach as the key will be same for all the Project Create / Update event, even if they are for different projects.

In case if I change the key to contain the project identifier also then it is not guaranteed to have all the project create event in single partition which indirectly means that validation operation may have to wait for long running create operation to finish.

How can I design my partitioning key?

Upvotes: 0

Views: 713

Answers (1)

Ran Lupovich
Ran Lupovich

Reputation: 1821

First of all you do not need to set your topic to work as compacted topic it is up to you to decide if that option is even relevant for your use case, it is configurable with the cleanup.policy topic parameter, secondly two use cases like you suggested on the same topic is not optimal and if you know it up front you might have easier time just setting two topics, having said that you can implement PartitionAssignor that will decide where to send each key, notice that the default assignor won't promise you that the two keys will go to different partitions, it just promises that all records with same key will go to same partition, there is no promise that each key will go to different partitions.

Hope that makes sense, please comment if further explanation is needed.

Upvotes: 2

Related Questions