CODEWITHSUNDEEP
mysqlapache-kafkadebezium
Bimal Kumar Dalei
Bimal Kumar Dalei

Reputation: 236

Some topics are unable to created as per tables of MySQL in Debezium

Am using Debezium for CDC from MySQL to Kafka topic, here problem is some topics are created and some topic are not created as per the lists of table in a particular database. I have 5 tables in the database and name is demo.

table_1, table_2, table_3, table_4 and table_5.

Able to create 2 tables like :

demo.demo.table_1
demo.demo.table_2

Expected topics should be :

 demo.demo.table_1
 demo.demo.table_2
 demo.demo.table_3
 demo.demo.table_4
 demo.demo.table_5

As per document of Debezium :

The default behavior is that a Debezium MySQL connector writes events for all INSERT, UPDATE, and DELETE operations in one table to one Kafka topic. The Kafka topic naming convention is as follows:

serverName.databaseName.tableName

Suppose that fulfillment is the server name, inventory is the database name, and the database contains tables named orders, customers, and products. The Debezium MySQL connector emits events to three Kafka topics, one for each table in the database:

fulfillment.inventory.orders
fulfillment.inventory.customers
fulfillment.inventory.products

Unable to find where exactly the issue is happening, and where should i have check for the problem ? Any log or permission issue.
This user has all permissions.

Below one is configuration .JSON file.

{
  "name": "inventory-connector",  
  "config": {  
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",  
    "database.hostname": "localhost",  
    "database.port": "3306",
    "database.user": "agent",
    "database.password": "123",
    "database.server.id": "123",  
    "database.allowPublicKeyRetrieval":"true",
    "database.server.name": "demo",  
    "database.include.list": "demo",  
    "database.history.kafka.bootstrap.servers": "localhost:9092",  
    "database.history.kafka.topic": "schema-changes.demo"  
  }
}

Please suggest me if anyone has any solution.

Thanks,
Bimal.

Upvotes: 1

Views: 1160

Answers (1)

Gitandi
Gitandi

Reputation: 46

{
  "name": "inventory-connector",  
  "config": {  
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",  
    "database.hostname": "localhost",  
    "database.port": "3306",
    "database.user": "agent",
    "database.password": "123",
    "database.server.id": "123",  
    "database.allowPublicKeyRetrieval":"true",
    "database.server.name": "demo",  
    "database.include.list": "demo",  
    "database.history.kafka.bootstrap.servers": "localhost:9092",  
    "database.history.kafka.topic": "schema-changes.demo",
    "name": "inventory-connector",
    "table.include.list": "table_1, table_2, table_3, table_4, table_5",
    "snapshot.mode": "schema_only"
  }
}

I think you missed "table.include.list" and by default debezium "snapshot.mode" is initial, it will get all the snapshot from the begining of the data in mysql binlog. if mysql binlog size is 1 GB, so you have to be patient to see the results, kafka broker still processing the log...

I suggest you to use "snapshot.mode": "schema_only", it will snapshot the schema only, and it will record only the data when the connector started.

And try to update the data from each table, the topics will generated automatically

I hope it works

regards

Upvotes: 2

Related Questions