CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connect
harry
harry

Reputation: 1281

Distributed Kafka Connect with multiple Connectors and one Topic

What's the behavior of Offset Management of a Kafka Connect cluster in Distributed mode, that is running multiple Connectors and listen to the same set of Topics (or one Topic)?

So in Distributed mode, Kafka Connect will store Offset information in Kafka, this Offset will be read and committed by the workers in the cluster. What happened if I have multiple Connectors running in that Kafka Connect cluster listening to the same Topic? Are the offset of a partition the same of all Connectors, or each Connector has different Offset on a partition?

Upvotes: 3

Views: 1897

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39810

Behind the scenes, sink connectors have consumers that belong to a consumer group (usually with group-id="connect-connectorName"). As far as I know, you cannot have two sink connectors within the same consumer group (there is no point of having two connectors in the same consumer group anyway - you can simply increase number of tasks if you want to). Therefore, in your use-case, when a message comes in, it will be processed by both connectors and two distinct offsets will be kept in __consumer_offsets topic (one per connector/consumerGroup).

Upvotes: 2

Related Questions