CODEWITHSUNDEEP
amazon-web-servicesaws-glue
Somnath Rokade
Somnath Rokade

Reputation: 665

Is it possible to add the data from multiple table of selective columns of mysql rds database to single table with another mysql rds table

Is it possible to add the data from multiple table of selective columns of mysql rds database to single table with another mysql rds instance using AWS Glue.

Please suggest.

Thanks

Upvotes: 1

Views: 820

Answers (1)

Prabhakar Reddy
Prabhakar Reddy

Reputation: 5144

Yes it is possible to achieve this with Glue via two approaches:

First approach:

  1. Run a Glue crawler on all these tables and load all these tables in to your Glue job from Glue catalog.

  2. Once you have loaded them in to multiple Glue DynamicFrames then you can select the columns along with the join key and join them.

  3. Then join these DynamicFrames and write the combined result back into MySql RDS table.

In this approach you will be loading all the columns from multiple tables and then selecting required columns inside your Glue job and join them.

Second Approach:

  1. You can frame a SQL query to select and join all these multiple tables and push it down to MySQL engine.

  2. The result is then calculated at MySQL engine and you will load this result into Spark DatFrame.

  3. Final step will be to convert this DataFrame to DynamicFrame and writing it to MySQL table.

In this approach you are delegating the computing task to MySQL and there will be impact on database engine if your tables are too big.

Upvotes: 1

Related Questions