CODEWITHSUNDEEP
aws-glue
Sherif Hamdy
Sherif Hamdy

Reputation: 105

Trigger Glue ETL jobs every 1 minute

I want to trigger a glue ETL job every 1 minute.

I found that there is a Schedule trigger but the minimum time is 5 minutes.

Is there any workaround for that? I can see that there is an On-demand trigger, can that be invoked from a lambda function?

Upvotes: 0

Views: 7642

Answers (2)

winnervc
winnervc

Reputation: 807

If that is not working for you, then there is a workaround using lambda function. You can create a lambda function which is triggered by a cloudwatch event (cron for evry 1 minute), using the boto3 module glue methodstart_job_run, here is the example of your syntax for lmabda function look like:

    import boto3
    glue= boto3.client('glue')
    def lambda_handler(event, context):    
        response = glue.start_job_run(JobName='string')

P.S:-- And also you need to have **

"concurrency limit" increase for your jobs.

** this can be acheivable only by contacting Amazon support and asking them to increase job concurrency, then only you can run multiple jobruns of a single job. because here you wants to triggere job for evry one minute which means while the other job run is in progress you need to have new job run (which is nothing but concurrent number of job runs)

Upvotes: 1

Yuriy Bondaruk
Yuriy Bondaruk

Reputation: 4750

You can do it if you are using CloudFormation templates where you can specify cron expression:

 Resources:
  YourJobNameTrigger:
    Type: AWS::Glue::Trigger
    Properties:
      Name: YourJobNameTriggerName
      Type: SCHEDULED
      Description: Daily scheduled trigger
      Schedule: cron(*/1 * * * ? *)
      Actions:
        - JobName: 
            Ref: YourJobName

Upvotes: -1

Related Questions