CODEWITHSUNDEEP
apache-kafka
magicduncan
magicduncan

Reputation: 1396

Marking Kafka entries to be excluded from log compaction

Knowing that Kafka is capable of "Log Compaction" by which it appears to guarantee that it will retain at least the latest entry for every known key, could this be extended to allow specified keys to be marked upfront as "special" such that they too would survive compaction even if they are no longer the latest entry for that key?

A use case might be a compacting historic log which can always give the head value for any given key, but in addition could allow specific historic values to be pinned for preservation.

Upvotes: 0

Views: 46

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

Currently (as of Kafka 0.10.0.1), this is not possible.

There is a new proposal (KIP-71) allowing to combine retention time and compaction for a single topic. This feature will be part of Kafka 0.10.1.0 which should be release in the next couple of weeks. Maybe this helps.

Upvotes: 3

Related Questions