CODEWITHSUNDEEP
parquetapache-kafka-connectdebeziums3-kafka-connector
shlomiLan
shlomiLan

Reputation: 716

Process messages with Null inside array in Kafka connect S3 connector

I'm using Kafka connect with 2 connectors:

  1. debezium to pull data from Postgres to Kafka
  2. S3 connector to save data from Kafka to S3

While running I got this error from the S3 connector

java.lang.NullPointerException: Array contains a null element at 0

I have found the related message that has the following as part of the message:

  "some_key": [
     "XCVB",
     null
  ]

How can I process this message?

I have tried adding the following to the S3 connector config:

"behavior.on.null.values": "ignore",
"errors.tolerance": "all",
"errors.deadletterqueue.topic.name":"dlq_s3_sink"

to try and skip those messages and send them to DLQ, but it doesn't seems to be working and the task failed on this error. I also saw this in the log:

Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.

but not sure where should I add this? as part of the connector config?

Upvotes: 2

Views: 513

Answers (1)

Akshay
Akshay

Reputation: 1

Add parquet.avro.write-old-list-structure:false in the sink connector config.

Also use Version 10.1.0 or above.
Reference: https://docs.confluent.io/kafka-connectors/s3-sink/current/changelog.html#version-10-1-0:~:text=S3%20transient%20errors-,PR%2D485%20%2D%20CCMSG%2D1531%3A%20Support%20null%20items%20within%20arrays%20with%20Parquet%20writer,-PR%2D475%20%2D%20CCMSG

Upvotes: 0

Related Questions