CODEWITHSUNDEEP
aws-glue
tommy vercetti
tommy vercetti

Reputation: 307

AWS Glue - starting ETL Job and pending execution

Sometimes when I want to run an ETL Job in AWS Glue, it is triggered immediately. But quite often it happens to me that it takes few minutes before ETL Job is doing anything - I see nothing in logs, only "pending execution". Is there any way I can influence it in configuration? Or is it totally up to AWS when it will start the job?

Upvotes: 6

Views: 5835

Answers (1)

user11985428
user11985428

Reputation: 81

I know this is an old thread, but I recently experienced the same symptoms. You'll have to confirm if the cause is/was the same. The job I originally created was to move data from my data lake in to a redshift db. When the job was created it had the redshift connection as a required resource, but it seems that glue was not able to acquire that resource prior to starting the job so the job would fail before it started every time... no logs.

I removed the connection as a requirement for the job by editing the job from the console. (Action->Edit job-> Required Connections->"X")

After I removed the requirement, the job ran just fine. It took me a long time to figure that one out. Hopefully it helps anyone else to runs into the same scenario.

Upvotes: 8

Related Questions