CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaaws-glueaws-step-functions
Sapnesh Naik
Sapnesh Naik

Reputation: 11636

Execute only one Glue job at a time / sequential Glue job execution

Currently, we have the following AWS setup for executing Glue jobs. An S3 event triggers a lambda function execution whose python logic triggers 10 AWS Glue jobs.

S3 -> Trigger -> Lambda -> 1 or more Glue Jobs.

With this setup, we see that at a time, multiple different Glue jobs run in parallel. How can I make it so that at any point in time, only one Glue job runs? And any Glue jobs sent for execution wait in a queue until the currently running Glue job is finished?

Upvotes: 2

Views: 2134

Answers (3)

Vasco Abreu
Vasco Abreu

Reputation: 11

Recently AWS released the "Job Run Queueing" feature, which lets you now take a simpler approach, which is to configure that job to have a Maximum concurrency of 1 and enable the "Job Run Queuing" option so that each run starts sequentially, if you're not interested in having control over the run sequence logic or you do not wish to resort to external services such as Step Functions.

https://aws.amazon.com/about-aws/whats-new/2024/09/aws-glue-job-queuing/

Upvotes: 0

Yuva
Yuva

Reputation: 3153

If you are looking for having some job queues to have the Glue jobs trigger in sequence, you may consider using a combination of SQS->lambda->Glue jobs? Please refer this SO for details

AWS Step function is also another option as suggested by Vaquar Khan

Upvotes: 2

vaquar khan
vaquar khan

Reputation: 11449

You can use step function and in each steps specify job you want to run so you will have control to run jobs and once step one complete then call step 2 jobs etc

Upvotes: 1

Related Questions