CODEWITHSUNDEEP
amazon-web-servicesamazon-sqsaws-step-functions
Sushil Verma
Sushil Verma

Reputation: 697

Polling SQS from Step function

I have two AWS Glue jobs in my application.

The first glue job produced some data and it passes to a lambda and lambda errors goes into a DLQ (SQS)

If there is no error messages in DLQ, I want to run the second Glue job.

I want to automate this logic with Step function. So the steps of step function will be:

  1. run first glue job.
  2. poll DLQ
  3. if DLQ has no message for some time, run second glue job.

Is there any way to poll SQS for messages from step function ?

Upvotes: 0

Views: 2874

Answers (1)

fedonev
fedonev

Reputation: 25639

Answering the main question: SFN cannot poll SQS directly. If you really need this kind of integration, SFN can invoke a Lambda to poll a queue and return the result back to your SFN.

[Edit: The recently introduced Step Function Service Integrations now allow calling the SQS SDK ReceiveMessage and DeleteMessage APIs on a Queue as a Step Function task. Although SQS polling is now available in SFN, Queues are arguably not the best fit for the scenario in the OP].

More generally, it's not a typical pattern to use queues as part of a StepFunction architecture. SFN orchestration is really an alternative to queue-based orchestration. SFN's core function is to orchestrate tasks and it has its own try-catch-error handling logic. Not that it can't be done with queues, just not sure what the benefit is.

A typical SFN integration pattern starts with an EventBridge cron that triggers a SFN execution. Glue Job 1 is the first task - SFN calls Glue's StartJobRun as a sync-type task and waits for the result. Chain together multiple lambda, glue event stages. Add Error branches to handle failure states. The job would end with a notification task.

Upvotes: 1

Related Questions