Reputation: 303
Im using Kafka and Debezium Postgres Connector.
I wanted to extract only record value without other details, so I am using the "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
SMT.
While this works for extracting the record value during inserts and updates, I've encountered an issue with delete events. After adding this SMT, I'm no longer receiving events when a record is deleted from the Postgres table.
Before adding the ExtractNewRecordState SMT, I would receive a tombstone message when a row was deleted, which I expected.
Is there a way to continue receiving tombstone messages after applying this transformation?
I tried with some smt but doesn't seem to work. here is the config:
{
"name": "debezium-postgres-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.dbname": "xxx",
"database.hostname": "xxx",
"database.password": "xx",
"database.port": "5432",
"database.server.name": "test-server",
"database.user": "xxxx",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": "true",
"plugin.name": "wal2json",
"publication.name": "dbz_publication",
"slot.name": "debezium_slot",
"table.include.list": "myTable",
"tasks.max": "1",
"transforms": "unwrap,ExtractField,ExtractKey",
"transforms.ExtractField.field": "data",
"transforms.ExtractField.type": "org.apache.kafka.connect.transforms.ExtractField$Value",
"transforms.ExtractKey.field": "key",
"transforms.ExtractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key", "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "xxxx",
"value.converter.schemas.enable": "true"
}
Upvotes: 0
Views: 311
Reputation: 303
Finally I found it, it's not very obvious in the doc https://debezium.io/documentation/reference/stable/transformations/event-flattening.html
I was using "delete.handling.mode" property but what I should use is the "transforms.unwrap.delete.handling.mode"
property since I am using the "transforms.unwrap" smt and set it to "none" to be able to receive tombstones messages.
There is also a note in the doc to use another property for the recent version.
This option is scheduled for removal in a future release. In its place, use the delete.tombstone.handling.mode option.
Upvotes: 1