CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebezium
Shubham Saroj
Shubham Saroj

Reputation: 404

What are the extra topics created when creating and debezium source connector

Q1) Following is my config which I used while creating the kafka connector for MySQL source.

{
  "connector.class": "io.debezium.connector.mysql.MySqlConnector",
  "snapshot.locking.mode": "minimal",
  "database.user": "cdc_user",
  "tasks.max": "3",
  "database.history.kafka.bootstrap.servers": "10.49.115.X:9092,10.48.X.211:9092,10.X.178.121:9092,10.53.4.X:9092",
  "database.history.kafka.topic": "history.cdc.fkw.supply.mp.seller_platform",
  "database.server.name": "cdc.fkw.supply.mp",
  "heartbeat.interval.ms": "5000",
  "database.port": "3306",
  "table.whitelist": "seller_platform.Contacts, seller_platform.EmailVerificationConfigs, seller_platform.financial_account_tag, seller_platform.HolidayConfigs, seller_platform.Preferences, seller_platform.Sellers",
  "database.hostname": "something.cloud.in",
  "database.password": "ABCDE",
  "database.history.kafka.recovery.poll.interval.ms": "5000",
  "name": "cdc.fkw.supply.mp.seller_platform.connector",
  "database.history.skip.unparseable.ddl": "true",
  "errors.tolerance": "all",
  "database.whitelist": "seller_platform",
  "snapshot.mode": "when_needed"
}


curl -s --location --request GET "http://10.24.18.167:80/connectors/cdc.fkw.supply.mp.seller_platform.connector/topics" | jq '.'
{
  "cdc.fkw.supply.mp.seller_platform.connector": {
    "topics": [
      "cdc.fkw.supply.mp.seller_platform.Sellers",
      "cdc.fkw.supply.mp",
      "cdc.fkw.supply.mp.seller_platform.HolidayConfigs",
      "cdc.fkw.supply.mp.seller_platform.EmailVerificationConfigs",
      "cdc.fkw.supply.mp.seller_platform.Contacts",
      "cdc.fkw.supply.mp.seller_platform.Preferences",
      "__debezium-heartbeat.cdc.fkw.supply.mp",
      "cdc.fkw.supply.mp.seller_platform.financial_account_tag"
    ]
  }
}

Why cdc.fkw.supply.mp and __debezium-heartbeat.cdc.fkw.supply.mp topic gets created?

I see some garbage data inside these 2 topics.

Q2)

Is there any rest api to know the kafka connect converter configuration on the worker server? If there is no API, then what is the path of the the configuration file where we store all worker properties?

This is the link of the worker properties: https://docs.confluent.io/platform/current/connect/references/allconfigs.html

curl -s --location --request GET "http://10.24.18.167:80"         
                                                            
{"version":"6.1.1-ccs","commit":"c209f70c6c2e52ae","kafka_cluster_id":"snBlf-kfTdCYWEO9IIEXTA"}%

Upvotes: 0

Views: 1327

Answers (1)

Shubham Saroj
Shubham Saroj

Reputation: 404

A1)

The heartbeat topic stores the details of all the kafka topics which the connector is using so that the connector can send heartbeat to it.

the database.server.name value named topic is created to store any schema changes that takes place in the database.

https://debezium.io/documentation/reference/1.7/connectors/mysql.html#mysql-schema-change-topic

Upvotes: 1

Related Questions