CODEWITHSUNDEEP
postgresqlapache-kafkaapache-kafka-connectdebezium
Oleksandr Klosovych
Oleksandr Klosovych

Reputation: 13

Debezium Outbox Pattern property transforms.outbox.table.expand.json.payload remove empty arrays

I'm implementing an outbox pattern using the debezium postgres connector, building up upon the official documentation: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html.

Evething ok but now I see strange behevior that empty array are missing in kafka messages if transforms.outbox.table.expand.json.payload tern to true

if in table I have value "{"some":"test", "agreemets":[]}" in topic message looks {"some":"test"}.

first line if we set to false second line if we set to true

enter image description here

How to prevent removeing empty arrays?

My full Kafka connect config

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "db",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname": "mycbh_dev",
    "database.server.name": "db",
    "schema.include.list": "common",
    "table.include.list": "common.outboxmessages",
    "tombstones.on.delete": "false",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "topic",
    "transforms.outbox.route.topic.replacement": "${routedByValue}",
    "transforms.outbox.table.field.event.timestamp": "created_at",
    "transforms.outbox.table.expand.json.payload": "true",
    "transforms.outbox.tracing.with.context.field.only": "true",
    "plugin.name": "pgoutput"
}

Upvotes: 1

Views: 944

Answers (1)

rastusik
rastusik

Reputation: 123

if you don't need to change anything inside the payload, you can just use this config debezium.format.value=simplestring ... This way you have to disable the json.expand.payload config, because there is no reason to use it anymore.

Upvotes: 0

Related Questions