CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Marco99
Marco99

Reputation: 1659

Distributed Kafka source connectors with multiple groups sharing same configurations

I have two Kafka connector nodes say Node-A and Node-B.

I run one distributed worker (pointing to the same Kafka cluster) in each node.

The group id and client id are unique across both the distributed workers.

Connector tasks are created successfully using http POST requests on both the nodes. The connector tasks produce messages for the same topic.

Note: I understand that the distributed workers are meant to have same group id for distribution of tasks. But I am doing as mentioned above for a specific case as an experiment.

I use the http GET request from Node-A say, as in http://localhost:8083/connectors

The http response (from Node-A) displays the connectors created from both the nodes i.e. from Node-A and Node-B, whereas I expect that the response should yield only the connectors created from Node-A and not those created from Node-B.

I want to know what is shared among these two distributed workers and how can i override this sharing?

Upvotes: 0

Views: 316

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191725

The http response (from Node-A) displays the connectors created from both the nodes i.e. from Node-A and Node-B

Sounds like both groups are sharing the same config and status topics.

status.storage.topic
offset.storage.topic
config.storage.topic

As per the documentation, it's recommended you read the description of what settings each topic should have, and create the topics manually

https://kafka.apache.org/documentation/#connect_running

Upvotes: 2

Related Questions