CODEWITHSUNDEEP
mysqlamazon-web-servicesapache-kafkaamazon-rdsaws-msk
Abdullah Khawer
Abdullah Khawer

Reputation: 5738

Create Kafka Connect Connector to Monitor AWS RDS MySQL Database in AWS MSK (Managed Streaming for Apache Kafka) Cluster

I have a use case where I have to capture data change in the AWS RDS MySQL database via Kafka connect and publish it on a Kafka topic. For Kafka, I'm using AWS Managed Streaming for Apache Kafka (MSK) service. I have created the topic. I'm just left with the connector part so that the consumers can see the messages/events while consuming a topic.

My question is, how can I create a Kafka Connect connector to Monitor AWS RDS MySQL Database in AWS Managed Streaming for Apache Kafka (MSK) cluster?

I was following Tutorial for Debezium 0.9 to achieve this using Docker containers for Kafka and AWS RDS instance. I want to achieve this using AWS Managed Streaming for Apache Kafka (MSK) instead of Docker containers. I just want to use Docker containers on the consumer side only.

Is this even possible or do I have to go with either AWS ECS or AWS EKS?

Upvotes: 2

Views: 2425

Answers (1)

Abdullah Khawer
Abdullah Khawer

Reputation: 5738

It is possible to create a Kafka Connect connector to monitor AWS RDS MySQL database by an AWS Managed Streaming for Apache Kafka (MSK) cluster by following the steps below:

  • AWS MSK cluster and AWS RDS MySQL database should exist in the same AWS VPC.
  • Security group for AWS MSK cluster should exist with inbound rule for AWS RDS MySQL database with the appropriate port.
  • Binary logging should be enabled and binlog_format should be set to ROW on the AWS RDS MySQL database.
  • User should be created in the AWS RDS MySQL database which will be used by the AWS MSK cluster to connect to the database with necessary privileges like SELECT and REPLICATION CLIENT.
  • Kafka Connect AWS plugin should be installed to connect to AWS MSK cluster. Download it as JAR file from the Confluent Hub and then copy it under the plugin.path directory in the Kafka Connect installation.
  • Kafka Connect connector configuration file should be created with the .properties extension with appropriate properties. A sample configuration file is below: 
    name=<CONNECTOR_NAME>
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=<RDS_MYSQL_HOSTNAME>
database.port=<RDS_MYSQL_PORT>
database.user=<RDS_MYSQL_USERNAME>
database.password=<RDS MySQL password>
database.whitelist=<LIST_OF_DATABASES_TO_MONITOR>
database.server.id=1
database.server.name=<DB_SERVER_NAME>
database.history.kafka.bootstrap.servers=<KAFKA_BROKER_ENDPOINTS>
database.history.kafka.topic=<KAFA_TOPIC_NAME>
  • Kafka Connect worker should be started using the following command: bin/connect-distributed.sh config/<CONNECTOR_CONFIGURATION_FILE_NAME>.properties
  • After that, send the Kafka Connect connector configuration by running the following command: curl -X POST -H "Content-Type: application/json" --data @<CONNECTOR_CONFIGURATION_FILE_NAME>.properties http://localhost:8083/connectors

Note: This is a generic solution and you can change it according to your use case. Also, don't forget to replace the following placeholders: <CONNECTOR_CONFIGURATION_FILE_NAME>, <CONNECTOR_NAME>, <RDS_MYSQL_HOSTNAME>, <RDS_MYSQL_PORT>, <RDS_MYSQL_USERNAME>, <RDS MySQL password>, <LIST_OF_DATABASES_TO_MONITOR>, <DB_SERVER_NAME>, <KAFKA_BROKER_ENDPOINTS>, and <KAFA_TOPIC_NAME> before proceeding.

References:

Upvotes: 1

Related Questions