Arpit Sharma
Arpit Sharma

Reputation: 910

Debezium OutboxEventRouter with json.expand.payload=true ignores empty arrays in sent kafka message

Debezium KafkaConnector tailing outbox_event table Table structure:

id (uuid)
key (varchar
topic (varchar)
payload (json)
emittedAt (timestampz)

KafkaConnector config:

config:
    database.dbname: ***
    database.hostname: ***
    database.password: ***
    database.port: ***
    database.user: ***
    decimal.handling.mode: double
    heartbeat.action.query: SELECT pg_logical_emit_message(false, 'heartbeat', now()::varchar);
    heartbeat.interval.ms: "300000"
    key.converter: org.apache.kafka.connect.storage.StringConverter
    key.converter.schemas.enable: "false"
    max.batch.size: "8192"
    max.queue.size: "13172"
    offset.flush.interval.ms: "15000"
    offset.flush.timeout.ms: "60000"
    plugin.name: pgoutput
    publication.autocreate.mode: filtered
    publication.name: dbz_publication_outbox
    schema.include.list: public
    signal.data.collection: public.dbz_signal
    slot.drop.on.stop: "false"
    slot.name: debezium_outbox
    snapshot.mode: never
    table.include.list: public.outbox_event
    time.precision.mode: connect
    tombstones.on.delete: "false"
    topic.prefix: outbox
    transforms: outbox
    transforms.outbox.route.by.field: topic
    transforms.outbox.route.topic.replacement: ${routedByValue}
    transforms.outbox.table.expand.json.payload: "true"
    transforms.outbox.table.field.event.key: key
    transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: "false"

Payload in DB

{
 "data": { "ids":[], "name":"name" }
}

Payload in Kafka Message

{
 "data": {"name":"name" }
}

The ids array is missing from the message. Any idea what might be going wrong.

I have tried setting probably: value.converter: org.apache.kafka.connect.string.StringConverter transforms.outbox.table.expand.json.payload: "true"

Upvotes: 0

Views: 139

Answers (0)

Related Questions