CODEWITHSUNDEEP
pythonpysparkaws-glue
Data Engineer
Data Engineer

Reputation: 67

Reusable AWS Glue Job

I have over 200 tables which need to be migrated to S3 from RDBMS with no transformations So we are planning to migrate using Glue Job. So I want to create AWS Glue Job which can be re-usable and executed using parameter values so that i can run for multiple tables at a time(Multi-threading). Is this possible anyway in AWS Glue.

Upvotes: 2

Views: 1471

Answers (1)

Emerson
Emerson

Reputation: 1166

The quick answer is yes

  1. You can reuse a singular glue job that can be resued where you can pass your source location and target database table name as job arguments to the glue job
  2. Your glue jobs supports concurrency which can be set in your glue job(whcih means you can have multiple invocations of the same job). This would be the easier option than implementing multi-threading in your job. But multi-threading would cretainly be possible as long as we use only default or pure python modules. There are certain account level limits (which can be increased) that you will need to keep in mind.
  3. You can pass the arguments to the glue job when you invoke the glue job, by using whatever mechanism you want (eg: step functions/lambdas...)

Upvotes: 3

Related Questions