Kafka Mirror Maker : Sync __consumer_offsets topic duplicates

Question

Following to the solution mentioned here kafka-mirror-maker-failing-to-replicate-consumer-offset-topic. I was able to start mirror maker without any error across DC1(Live Kafka cluster) and DC2(Backup Kafka cluster) clusters.

Looks like it is also able to sync __consumer_offsets topic across DC2 cluster form DC1 cluster.

Issue

If I close down consumer for DC1 and point same consumer(same group_id) to DC2 it reads the same messages again even though mirror maker is able sync offsets for this topic and partitions.

I can see that LOG-END-OFFSET is showing correctly but CURRENT-OFFSET is still pointing to old causing LAG.

Example

• Mirror Maker is still running in DC2.

• Before consumer shut down in DC1

  //DC1  __consumer_offsets topic
  +-----------------------------------------------------------------+
  | TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  |
  +-----------------------------------------------------------------+
  | gs.suraj.test.1 0          10626           10626           0    |
  | gs.suraj.test.1 2          10619           10619           0    |
  | gs.suraj.test.1 1          10598           10598           0    |
  +-----------------------------------------------------------------+
  

• Stop consumer in DC1

• Before consumer start up in DC2

  //DC2  __consumer_offsets topic
  +-----------------------------------------------------------------+
  | TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  |
  +-----------------------------------------------------------------+
  | gs.suraj.test.1 0          9098            10614           1516 |
  | gs.suraj.test.1 2          9098            10614           1516 |
  | gs.suraj.test.1 1          9098            10615           1517 |
  +-----------------------------------------------------------------+
  

Because of this lag, when I start same consumer in DC2 in reads 4549 messages again, which should not happen as it is already read an commited in DC1 and mirror maker have sync __consumer_offsets topic from DC1 to DC2

Please let me know if I am missing anything in here.

Answer 1

If you are using Mirror Maker 2.0 they say explicitly on the motivation that there is no support for exactly-once:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Motivation

But they intend to do it in the future.