MirrorMaker2 does not mirror all messages from a compacted topic?

Question

In some of the mirrored topics I'm missing data in the mirrored sink topic!

The source Kafka cluster is configured by default with log.rentention.ms=-1 and log.cleanup.policy=compact.

Pattern of usage: records are inserted and the with the key are also been nulled (so Kafka is used somehow like a database)

MM2 Config:

• sourceCluster config: (receive.buffer.bytes: 8192; max.poll.records: 10000)
• sinkCluster config: (batch.size:8192; offset.flush.timeout.ms:1200000; connection.max.idle.ms: 18000000)

following metrics are available for old and the new sink topic:

• Topic X: not nulled msg count: 63k new: 16k (I've also checked the head and tail there are the same creationTime)
• Topic Y: not nulled msg count: 30k new 18k; same story with the head/tail
• Topic Z: not nulled msg count: 136 new 136; SAME msg count

My is assumption is that when MM2 is consuming a batch and when it this batch same key is being nulled the first msg is not been mirrored. Is this correct 🤔? If this is assumption is right how would one mirror an event-sourcing topic, If you want to recreated a state exactly in the same way how it is in the source topic?

Why messages are missing in the mirrored sink topic? How can I mirror all messages from the source topic?

Answer 1

I’ve writing a blogpost about the issue and script help comparing the result of both topics https://imalik8088.de/posts/kafka_where_are_all_my_messages/