CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqsaws-step-functions
james
james

Reputation: 418

Preserve FIFO processing with an AWS SQS FIFO Queue as a lambda trigger event source and Step function processing

Is it possible through configuration to setup a SQS FIFO queue with a lambda trigger attached to use the lambda to call a step function while preserving order all the way through processing? So this would mean not firing off multiple step functions in parallel in the running state, but preserving the FIFO processing so any new step function executions would need to wait until the step function completes before processing the next batch of records.

Is this something I can achieve through configuration or is another route necessary ?

Upvotes: 0

Views: 489

Answers (2)

user22896388
user22896388

Reputation:

Yes. Something like this may work:

Use your Lambda trigger to prepare the inputs for a Step Function that will serve as a "launcher." Start the launcher with this input as an array and be sure to include the recipient handles for each of the FIFO queue messages. (Note: You won't delete the messages from within the trigger. This will keep the queue messages in flight and no other messages from the message group will be receieved by the trigger.)

Run the launcher. Iterate over the array of input. Start the other step functions.

(Tip: If you structure the contents of the array of input in a way where each array value is another array of messages within each message group, then you may process the message groups in parallel).

When each Step Function is done, delete the FIFO messages from the queue using the recipient handle that you sent along with your input. (Tip: You may want to catch the Step Functions' errors so that you can do whatever you may want to do with failed messages).

Upvotes: 0

Chris Armstrong
Chris Armstrong

Reputation: 61

I don't think this would be possible if you're using the builtin integration between SQS and Lambda - as soon as your function execution exits (which will be as soon as it is finished starting the step function), AWS Lambda will fetch the next message.

If your execution time is relatively short, and you are willing to pay the Lambda execution time, you could poll or wait on the Step Function to complete in your Lambda function before returning. This could be expensive however if you have a lot of messages to process or your Step function takes some time to complete.

Upvotes: 1

Related Questions