CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectaws-mskapache-kafka-mirrormaker
Scourrge
Scourrge

Reputation: 103

Kafka Mirrormaker 2 replication from latest offset instead of earliest

I have a problem with replication of topics from cluster A to cluster B with Mirrormaker 2. I want to replicate these topics from latest offset instead of replicating them from the beginning. But when I set auto.offset.reset=latest nothing happens and replication still happens from the beginning of topics instead of an end. Do I set it wrong?

This is my whole MM2 configuration:

clusters = A, B

A.bootstrap.servers = broker-01A:9092
B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092

replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

#Enable replication between clusters and define topics which should be replicated
A->B.enabled = true
A->B.topics = .*
A->B.replication.factor=3
A->B.emit.heartbeats.enabled = true
A->B.emit.checkpoints.enabled = true

auto.offset.reset=latest
refresh.topics.enabled=true

heartbeats.topic.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1

config.storage.replication.factor = 1
offset.storage.replication.factor = 1
status.storage.replication.factor = 1

UPDATE

After changing the auto.offset.reset=latest to A.consumer.auto.offset.reset=latest the auto.offset.reset=latest configuration started to appear in Mirrormaker logs. But still no luck. Still mirroring is done from the start.

Upvotes: 2

Views: 5264

Answers (1)

Patrik
Patrik

Reputation: 31

Not a fix but found this (currently open) issue that might be responsible: https://issues.apache.org/jira/browse/KAFKA-13988

Upvotes: 1

Related Questions