CODEWITHSUNDEEP
mysqlapache-kafkaapache-kafka-connectamazon-auroracdc
Anupam
Anupam

Reputation: 274

How to connect MSK from Aurora using open source kafka connect

I have to put my CDC changes from Aurora mysql db to MSK kafka topic . I think there is no inbuilt mechanism in aws like DMS which will sink all CDC from Aurora to MSK topic and in their docs they have not mentioned any support for that .

So i am only left with using something open source . To use that i have few questions around that please help me with this .

  1. I was doing some search and everywhere i see kafka-connect-jdbc for this.So first this is this open source free licence ?
  2. I have seen debezium one as well which connects mysql to msk.

  3. Aurora record will be text and i need to put record into MSK as JSON ,so i have to use schema registry ? is schema registry free licence or open source or comes MSK kafka ?

  4. If i have to use kafka -connect from confluent or debezium i need EC2 instance .In this what i need to install ? Only kafka or with kafka confluent and debezium as well .

Please suggest something which is free licence and open source .

Upvotes: 0

Views: 2429

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191864

1) If you want to use Confluent Platform components outside of Zookeeper, Apache Kafka brokers and base Kafka Connect (such as the JDBC connector plugin) please read https://www.confluent.io/confluent-community-license-faq/

2) Debezium should work fine. It's under the Apache 2.0 License ; you can use Apache Kafka with it, not the rest of the Confluent Platform

3a) Schema Registry is only capable of storing Avro, not JSON. Therefore you don't require it. 3b) See 1 for Schema Registry licensing

4) You will need EC2 or use Docker Kafka Connect / Debezium containers via ECS / EKS.

There are still other options available capable of CDC into Kafka, some of which I know are open source or even commercial supported, but listing them here is too broad.

Regarding built in AWS services, you can trigger lambda functions on Aurora tables to do whatever you want

https://aws.amazon.com/blogs/database/capturing-data-changes-in-amazon-aurora-using-aws-lambda/

It might be possible Lambda has / will eventually have MSK integration

Upvotes: 0

Related Questions