CODEWITHSUNDEEP
apache-kafka
Zazaeil
Zazaeil

Reputation: 4109

Custom compaction for Kafka topic on the broker side?

Assume some Kafka cluster with some topic named MyTopic. According to business logic I am implementing, adjancent records are considered equal whenever some subset of value's rather then key's properties are equal. Thus, built-in compaction, driven by key equality, doesn't work for my scenario. I could implement pseudocompaction at the consumer side, which is neither an option due to performance. The whole idea is to maintain right compaction at the broker side. In addition to that, such a compaction has to be applied only within some special consumer group; all other groups have to get entire log of records as they are now.

According to my knowledge there is no way to implement such compaction. Am I wrong?

Upvotes: 1

Views: 752

Answers (2)

Mickael Maison
Mickael Maison

Reputation: 26865

This question has already been answered correct, ie it's not currently possible. But it's worth noting that KIP-280 has been approved and will add new compaction policies. It is currently targeted for Kafka 2.5.

It looks like your goal would be achieved with the new header policy.

Upvotes: 2

Oras
Oras

Reputation: 1096

You can not have custom log compaction. It is either delete or compact based on keys. https://kafka.apache.org/documentation/#compaction

However, if your case is just related to some special consumer groups, you might create a stream to read your specified topic, create a hash key (based on value subset) which will write to another topic and apply clean up policy compaction to this new topic.

This obviously will have almost duplicated data which might not suit your case.

Upvotes: 3

Related Questions