CODEWITHSUNDEEP
apache-kafkaapache-flink
user2968134
user2968134

Reputation: 11

In apache flink with kafka,How to connect to multiple kafka cluster with same group id

We are using Apache flink with Kafka. Our flink jobs deployed in 3 datcenter dc1, dc2, dc3. In each data centre, flink job consume messages from their respective data centre kafka cluster.

Now we have changed the design that each flink job cross connects to all 3 data centre of kafka cluster with same group id, but we are receiving duplicates even passing same group id? So how to avoid this duplicates

Wanted to know how to avoid duplicate messages while connecting to multiple clusters of kafka?

Upvotes: 0

Views: 492

Answers (1)

Martijn Visser
Martijn Visser

Reputation: 2068

It's not entirely clear if you're having one Kafka cluster, stretched across 3 data centers, or three Kafka clusters, each running in their own DC.

If you have one Kafka cluster which is stretched across 3 DCs, then you also need to setup one Flink cluster and job which accesses the 3 DCs. If you have three Kafka clusters, then you should also have 3 Flink clusters and jobs.

Upvotes: 0

Related Questions