Sara M.
Sara M.

Reputation: 303

How to receive tombstone messages in debezium connector?

Im using Kafka and Debezium Postgres Connector. I wanted to extract only record value without other details, so I am using the "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState" SMT.

While this works for extracting the record value during inserts and updates, I've encountered an issue with delete events. After adding this SMT, I'm no longer receiving events when a record is deleted from the Postgres table.

Before adding the ExtractNewRecordState SMT, I would receive a tombstone message when a row was deleted, which I expected.

Is there a way to continue receiving tombstone messages after applying this transformation?

I tried with some smt but doesn't seem to work. here is the config:

{
"name": "debezium-postgres-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.dbname": "xxx",
"database.hostname": "xxx",
"database.password": "xx",
"database.port": "5432",
"database.server.name": "test-server",
"database.user": "xxxx",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": "true",
"plugin.name": "wal2json",
"publication.name": "dbz_publication",
"slot.name": "debezium_slot",
"table.include.list": "myTable",
"tasks.max": "1",
"transforms": "unwrap,ExtractField,ExtractKey",
"transforms.ExtractField.field": "data",
"transforms.ExtractField.type": "org.apache.kafka.connect.transforms.ExtractField$Value",
"transforms.ExtractKey.field": "key",
"transforms.ExtractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key", "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "xxxx",
"value.converter.schemas.enable": "true"
}

Upvotes: 0

Views: 311

Answers (1)

Sara M.
Sara M.

Reputation: 303

Finally I found it, it's not very obvious in the doc https://debezium.io/documentation/reference/stable/transformations/event-flattening.html

I was using "delete.handling.mode" property but what I should use is the "transforms.unwrap.delete.handling.mode" property since I am using the "transforms.unwrap" smt and set it to "none" to be able to receive tombstones messages.

There is also a note in the doc to use another property for the recent version.

This option is scheduled for removal in a future release. In its place, use the delete.tombstone.handling.mode option.

Upvotes: 1

Related Questions