CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Prerana Jain
Prerana Jain

Reputation: 151

How to run Kafka-connect-replicator in distributed mode?

I want to replicate data from one system to another using confluent's replicator.I am using two Ubuntu 18.04 systems where one is acting as source and other as destination.

I tried to run kafka-connect-replicator in distributed mode where I changed the following configurations:

  1. In confluent/etc/kafka/server.properties I made the following changes 
SOURCE
> advertised.listeners=PLAINTEXT://source.ip:9092
DESTINATION
> advertised.listeners=PLAINTEXT://destination.ip:9092
  1. In confluent/etc/kafka-connect-replicator/replicator.connect.distributed.properties I made the following changes 

 - group.id=connect-replicator

group.id is same on source and destination system

SOURCE

 - bootstrap.servers=destination.ip:9092, source.ip:9092

DESTINATION

 - bootstrap.servers=destination.ip:9092, source.ip:9092
  1. In confluent/etc/kafka-connect-replicator/quickstart-replicator.properties I changed the following configurations

SOURCE

name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
# source cluster connection info
src.kafka.bootstrap.servers=source.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the source
src.zookeeper.connect=localhost:2181

# destination cluster connection info
dest.kafka.bootstrap.servers=destination.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the destination
dest.zookeeper.connect=destination.ip:2181
# configure topics to replicate
topic.whitelist= test-topic
topic.rename.format=${topic}.replica

DESTINATION

name=replicator-source
connector.class=io.confluent.connect.replicator.ReplicatorSourceConnector
# source cluster connection info
src.kafka.bootstrap.servers=source.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the source
src.zookeeper.connect=source.ip:2181

# destination cluster connection info
dest.kafka.bootstrap.servers=destination.ip:9092
# Set to use direct connection to Zookeeper by Replicator on the destination
dest.zookeeper.connect=destination.ip:2181
# configure topics to replicate
topic.whitelist= test-topic
topic.rename.format=${topic}.replica

And then I created topic in source system and launched the connector using the below command

PATH_TO_CONFLUENT> sudo ./bin/connect-distributed ./etc/kafka-connect-replicator/replicator-connect-distributed.properties ./etc/kafka-connect-replicator/quickstart-replicator.properties

After this I produced data in the topic from source system and try to consume in destination system with the topic name {topic}.replica but there is not topic present to consume from.

Upvotes: 0

Views: 1176

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

It's not clear what errors you've having, but some notes.

  1. connect-distributed only takes one property file, not two. You HTTP Post the Properties to the Connect Cluster as JSON, not load a properties file during cluster startup. The quickstart file is meant to be used for connect-standalone

The JSON would look like 

{"name": "your-replicator-name", "config": {
  "src.kafka.bootstrap.servers": "...",
  ...
}

  1. ./etc/kafka/connect-distributed.properties should be a starting point for running any Connect or Replicator cluster in Distributed mode, although there may be similar configurations in replicator-connect-distributed.properties

  2. bootstrap.servers should only ever point to a single cluster. The source and destination would be separated within src.kafka.bootstrap.servers and dest.kafka.bootstrap.servers

Upvotes: 1

Related Questions