CODEWITHSUNDEEP
pythonamazon-web-servicesaws-glue
Jsimp
Jsimp

Reputation: 58

How to sync an entire data catalog with RDS

My goal is to always automatically replicate an S3 bucket with a schema in RDS (MSSQL).

I have a Glue crawler that is triggered by a lambda function on every S3 upload. Hence, the data catalog is always in sync with S3. I am able to create Glue jobs that automatically uploads a specific table from the catalog. If a user uploads a new table with new columns, I have to manually create a new job for that table. Is there a way to automate this manual step?

Upvotes: 0

Views: 258

Answers (1)

JD D
JD D

Reputation: 8137

If you want to trigger some logic when a new table is discovered by your crawler, I would suggest you look into creating a Cloudwatch Event Rule for a state change in the Glue service. Specifically, you can invoke a service (i.e. a Lambda function) when the state of a Data Catalog database or table changes.

This lambda function can inspect the state change event and use one of the SDKs to create the job for the new table.

Upvotes: 0

Related Questions