CODEWITHSUNDEEP
apache-kafka
Vish
Vish

Reputation: 847

LogCompaction and consumer in Kafka

This question is about the log compacted topics. I read through the log tail and log head concepts in the documentation. What I am not sure of is what triggers the log tail to grow or compaction to occur. For e.g. if a client is processing messages off a topic and is taking it’s own sweet time to do so and when it is processing, 5 messages with the key “a” are delivered to the topic. Would the slow client receive all 5 messages or can compaction occur in the meanwhile and the client end up receiving only the last message for key “a”. IOW, will log tail compact messages which haven’t been processed by clients that re currently connected?

Upvotes: 0

Views: 36

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

Given the default segment size of 1GB and message size of 1MB, and a fresh set of keyed messages, your consumer will see all 5 "a" keys because given those values, you would still need to produce at least 995MB worth of data

Only closed segments (files where your data is stored) are compacted, and this happens on a scheduled LogCleaner thread. You can tweak the dirty ratio and/or segment size of the topic to make sure compaction is more frequent, at the risk of IO pressure on the broker

Upvotes: 2

Related Questions