CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Jeffy Zhang
Jeffy Zhang

Reputation: 111

how to dynamically update topics of SinkConnector in Kafka Connect?

I have written a Kafka Connect to consumer topics, but my topic will change at runtime, so I need reconfiguration the topics.

I know that use RESTful API can update topics is there another way to achieve this?

Upvotes: 4

Views: 6829

Answers (2)

gaurav dwivedi
gaurav dwivedi

Reputation: 36

Kafka Connect is intended to be run as a service, it also supports a REST API for managing connectors. Only way to update this through REST API at run time :

PUT /connectors/{name}/config - to update the connector's configuration parameters at run time.

Request Json Object - config(map)

{
    "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
    "tasks.max": "20",
    "topics": "kafkaConnectTopic",
    "hdfs.url": "hdfs://smoketest:9000",
    "hadoop.conf.dir": "/etc/hadoop/conf",
    "hadoop.home": "/etc/hadoop",
    "flush.size": "1000",
    "rotate.interval.ms": "100"
}

Response :

{
    "name": "hdfs-sink-connector",
    "config": {
        "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
        "tasks.max": "20",
        "topics": "kafkaConnectTopic",
        "hdfs.url": "hdfs://smoketest:9000",
        "hadoop.conf.dir": "/etc/hadoop/conf",
        "hadoop.home": "/etc/hadoop",
        "flush.size": "1000",
        "rotate.interval.ms": "100"
    },
    "tasks": [
        { "connector": "hdfs-sink-connector", "task": 1 },
        { "connector": "hdfs-sink-connector", "task": 2 },
        { "connector": "hdfs-sink-connector", "task": 3 }
   ]
}

for further reading you can go through http://docs.confluent.io/3.0.0/connect/userguide.html#connect-administration.

Upvotes: 2

dawsaw
dawsaw

Reputation: 2313

You can specify a list of topics to consume from in the connector configuration if you know ahead of time the set of topics you will want to switch to. Otherwise, the the REST API is the only way to update configurations on the fly.

Upvotes: 0

Related Questions