CODEWITHSUNDEEP
amazon-web-servicesamazon-rdsaws-glueopensearch
user16012143
user16012143

Reputation: 131

RDS(dynamic schema) -> AWS opensearch by using AWS Glue

I am using AWS RDS(MySQL) and I would like to sync this data to AWS elasticsearch in real-time.

I am thinking that the best solution for this is AWS Glue but I am not sure about I could realize what I want.

This is information for my RDS database:
■ RDS
・I would like to sync several tables(MySQL) to opensearch(1 table to 1 index).
・The schema of tables will be changed dynamically.
・The new column will be added or The existing columns will be removed since previous sync.
(so I also have to sync this schema change)

Could you teach me roughly whether I could do these things by AWS Glue?

I wonder if AWS Glue can deal with dynamic schame change and syncing in (near) real-time.
Thank you in advance.

Upvotes: 0

Views: 1909

Answers (1)

Sudarshan kumar
Sudarshan kumar

Reputation: 1585

Glue Now have OpenSearch connector but Glue is like a ETL tool and does batch kind of operation very well but event based or very frequent load to elastic search might not be best fit ,and cost also can be high . https://docs.aws.amazon.com/glue/latest/ug/tutorial-elastisearch-connector.html

DMS can help not completely as you have mentioned schema keeps changing .

Logstash Solution

Since Elasticsearch 1.5, Elasticsearch added jdbc input plugin in Logstash to sync MySQL data into Elasticsearch.

AWS Native solution

You can have a lambda function on MySQL event Invoking a Lambda function from an Amazon Aurora MySQL DB cluster

The lambda will write to Kinesis Firehouse in json and kinesis can load into OpenSearch .

Upvotes: 1

Related Questions