CODEWITHSUNDEEP
apache-kafkaapache-kafka-connectconfluent-schema-registrydebezium
kalyan chakravarthy
kalyan chakravarthy

Reputation: 653

Kafka Connect with Debezium is creating schemas for Key and Value under same schema

Am using Debezium MySQL connector to stream changes from database to Kafka. I want to use AWS Glue schema registry to store the schemas. Below configuration is creating the schemas for Key and Value in same schema name "database1.tutorial.movies" under the registry "msk-cdc" as two different versions. By default am expecting different schemas to be created for with "database1.tutorial.movies-Key" and "database1.tutorial.movies-Value". What am I doing wrong here?

name=debezium-mysql-sample
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=<hostname>
database.port=3306
database.user=<username>
database.password=<passowrd>
database.server.id=42
database.server.name=database1
table.whitelist=tutorial.movies
database.history.kafka.bootstrap.servers=<bootstrap-server>
database.history.kafka.topic=dbhistory.demo1
key.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
key.converter.compressionType=NONE
value.converter.compressionType=NONE
key.converter.endpoint=https://glue.us-east-1.amazonaws.com
value.converter.endpoint=https://glue.us-east-1.amazonaws.com
key.converter.region=us-east-1
value.converter.region=us-east-1
key.converter.timeToLiveMillis=3600000
value.converter.timeToLiveMillis=3600000
key.converter.cacheSize=100
value.converter.cacheSize=100
key.converter.avroRecordType=GENERIC_RECORD
value.converter.avroRecordType=GENERIC_RECORD
key.converter.registry.name=CDC
value.converter.registry.name=CDC
key.converter.compatibility=NONE
value.converter.compatibility=NONE
key.converter.description=none
value.converter.description=none
key.converter.schemaAutoRegistrationEnabled=true
value.converter.schemaAutoRegistrationEnabled=true
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.drop.tombstones=false
transforms.unwrap.delete.handling.mode=rewrite
transforms.unwrap.add.fields=op,source.ts_ms

Upvotes: 0

Views: 929

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191671

You might want to look at setting a specific naming strategy.

By default, it appears to only use the topic name, and doesn't differentiate between keys and values.

Related issue - https://github.com/awslabs/aws-glue-schema-registry/issues/93
Serializer Source - https://github.com/awslabs/aws-glue-schema-registry/blob/master/serializer-deserializer/src/main/java/com/amazonaws/services/schemaregistry/serializers/avro/AWSKafkaAvroSerializer.java#L148

Also, your record keys should ideally just be the primary key type of the database, so String, Int, or Long Converter would be more appropriate than Avro, in that case

Upvotes: 0

Related Questions