CODEWITHSUNDEEP
javamysqltimestampapache-kafka-connectdebezium
user3865748
user3865748

Reputation: 51

Kafka Connect Debezium MySQL Connector Timestamp Issue

I'm using Debezium MySQL Connector deployed in Kafka Connect to Stream MySQL Changes to Kafka Topics and from where we fetch these messages enrich data and push the data to another MySQL.

Both source and sink are MySQL.

I have couple of Columns in my source table with column datatype as TIMESTAMP.

Ex : create_time : 2021-10-06 09:32:46

I could see the above data in Kafka Message as below

"create_time":"2021-10-06T09:32:46Z"

Facing Issue with T and Z in the above String when the message is loaded into target database (MySQL),where the data type is TIMESTAMP as well.

I need create_time in kafka message as "create_time":"2021-10-06 09:32:46" without TZ

Error log :

I Understand that Debezium is converting Timestamp column data to ZonedDateTime which might be adding the TimeZone Information. (https://debezium.io/documentation/reference/1.7/connectors/mysql.html#mysql-temporal-types)

I tried org.apache.kafka.connect.transforms.TimestampConverter in Debezium Connector Configurations but this is not helping.

Currently I'm using org.apache.kafka.connect.json.JsonConverter in Kafka Connect Configurations.

Things to Note : I'm batching multiple messages received from Kafka, Construct a csv file and by using LOAD DATA I'm pushing the csv contents to Target database.

Upvotes: 0

Views: 2041

Answers (1)

DoroCoder
DoroCoder

Reputation: 111

Your attempt using 'org.apache.kafka.connect.transforms.TimestampConverter' seemed correct in direction but not usage. The timestamp converting object 'in a sink connector' must contain its 'type', 'field', 'format' and 'target.type' very very correctly for each targeted column. I'm very happy with this feature in my official serial projects. ;-)

Upvotes: 0

Related Questions