CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectdebezium
Alexey
Alexey

Reputation: 1434

Can I initiate an ad-hoc Debezium snapshot without a signaling table?

I am running a Debezium connector to PostgreSQL. The snapshot.mode I use is initial, since I don't want to resnapshot just because the connector has been restarted. However, during development I want to restart the process, as the messages expire from Kafka before they have been read.

If I delete and recreate the connector via Kafka Connect REST API, this doesn't do anything, as the information in the offset/status/config topics is preserved. I have to delete and recreate them when restarting the whole connect cluster to trigger another snapshot.

Am I missing a more convenient way of doing this?

Upvotes: 0

Views: 1591

Answers (2)

maruthisi
maruthisi

Reputation: 1

The answer may be late, but I too had this requirement. Posting it to help someone in future.

I configured the "snapshot.mode" as "always" in the dabezium-postgres.json file. So everytime I deleted and added this dabezium connector (via DELETE and POST REST API respectively) it triggers a full snapshot with READ events. Here the kafka-connect server is not restarted. My application (kafka consumer) is reentrant to these events and processes them.

Upvotes: 0

stolen_leaves
stolen_leaves

Reputation: 1462

You will also need a new name for the connector as well as a new database.server.name name in the connector config, which stores all the offset information. It should almost be like deploying a connector for the first time again.

Upvotes: 1

Related Questions