How to deserialize the avro data generated by debezium correctly

Question

I'm using debezium to capture data change from Mysql, the connect configuration is:

{
    "name": "avro-mysql-cdc-payments-connector",
    "config": {
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "key.converter.schemas.enable": "false",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter.schemas.enable": "false",
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
        "tasks.max": "1",
        "database.server.id": "100001",
        "database.hostname": "mysql",
        "database.port": "3306",
        "database.user": "debezium",
        "database.password": "debezium",
        "topic.prefix": "avro.mysql",
        "database.include.list": "cdc",
        "table.include.list": "cdc.payments",
        "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
        "schema.history.internal.kafka.topic": "avro.schema-changes.mysql.cdc.payments",
        "snapshot.locking.mode": "none",
        "key.converter.schema.registry.url": "http://schema-registry:8081",
        "value.converter.schema.registry.url": "http://schema-registry:8081"
    }
}

and in the schema registry, the time colums have mapped to:

Type in Mysql	Type in schema registry
date	{"type": "int32","optional": false,"name": "io.debezium.time.Date","version": 1,"field": "created_date"}
timestamp(6)	{"type": "string","optional": false,"name": "io.debezium.time.ZonedTimestamp","version": 1,"default": "1970-01-01T00:00:00Z","field":"updated_time"}
datetime(6)	{"type": "int64","optional": false,"name": "io.debezium.time.MicroTimestamp","version": 1,"default": 0,"field": "created_datetime"}

Then I wrote code to deserialize the avro data by refering avro_consumer.py Those time columns can't decoded to python datetime object, but are the same types as in the schema registry, namely string/int.

Is there a way to correctly decode these types, or does debezium provide corresponding decoding plugins?

Code to decode debezium avro data, either Python or Scala is fine.

Answer 1

can't decoded to python datetime object

Avro has no native Date type, only int/string. You'll need to parse in your consumer, accordingly