CODEWITHSUNDEEP
apache-kafka
redboy
redboy

Reputation: 607

Kafka MirrorMaker replication between multiple regions

NOTE: Also posted to [email protected].

I need a solution that will mirror a topic to a remote location that avoids cycles and a topic per region issue.

This solution would require EXACTLY two topics regardless of how many regions (datacenters) it replicates to.

I would like the initial producer (P1) to write to only a local (L1) and remote topic (R1). I would like MirrorMaker (M1) to read from the remote topic (R1) and write to the other regions local topic (L2). The other region in kind would write using (P2) to its local (L2) and remote (R2) and a separate MirrorMaker (M2) would replicate its remote (R2) back to the other regions local (L1).

If I add a third region, I would just need to add another MIrrorMaker to existing regions and another to replicate from third back to existing.

------------------------------- 
REGION TOTAL: 2
------------------------------- 
P1 -> L1, R1 
P2 -> L2, R2

M1 (R1 -> L2) 
M2 (R2 -> L1)

------------------------------- 
REGION TOTAL: 3
------------------------------- 
P1 -> L1, R1 
P2 -> L2, R2 
P3 -> L3, R3

M1.1 (R1 -> L2) 
M1.2 (R1 -> L3)

M2.1 (R2 -> L1) 
M2.2 (R2 -> L3)

M3.1 (R3 -> L1) 
M3.2 (R3-> L2)

Upvotes: 1

Views: 618

Answers (1)

redboy
redboy

Reputation: 607

Here is a solution.

Each region writes to its own topic. MirrorMaker replicates each region's specific topic to every other region. The consumer in all regions consumes from all three topics.

This solution requires one write per producer instead of two. N topics (1 per region) and N(N-1) mirror makers and N consumers consuming N topics each.

Upvotes: 1

Related Questions