CODEWITHSUNDEEP
apache-kafka
RamPrakash
RamPrakash

Reputation: 3234

Kafka - Log compaction behavior

I am trying to understand the log compaction.

For a topic, my settings are

min.cleanable.dirty.ratio   0.005
segment.ms  5000
cleanup.policy  compact

I produced below messages to the topic

a   0
b   1
c   1
f   4
r   0
s   1
u   1
f   5
h   1

I am trying to consumer via kafka console consumer (even after waiting 5 mins and created a new console consumer) - i still see the 2 occurrences of f 

a   0
b   1
c   1
f   4
r   0
s   1
u   1
f   5
h   1

Should it not be removed?

Upvotes: 3

Views: 481

Answers (1)

Bartosz Wardziński
Bartosz Wardziński

Reputation: 6583

Only messages that are not in active segment can be consider in compaction process. Even if you set segment.ms= 5000, rolling new log segment can be made when new messages for partition appear.

If you send all message at once, all messages will go to the same segment and that segment will be active. It became valid for compaction if new messages will arrive after 5000ms. Log cleaner after log.retention.check.interval.ms will try to compact messages with same key.

Upvotes: 4

Related Questions