CODEWITHSUNDEEP
amazon-web-servicesaws-step-functions
TimJohnson
TimJohnson

Reputation: 933

How do you run functions in parallel?

My desire is to retrieve x number of records from a database based on some custom select statement, the output will be an array of json data. I then want to pass each element in the array into another lambda function in parallel.

So if 1000 records are returned, 1000 lambda functions need to be executed in parallel (I increase my account limit to what I need). If 30 out of 1000 fail, the main task that was retrieving the records needs to know about it.

I'm struggling to put together this simple flow.

I currently use javascript and AWS Aurora. I'm not looking for node.js/javascript code that retrieves the data, just the AWS Step Functions configuration and how to build an array within each function.

Thank you.

Upvotes: 1

Views: 306

Answers (1)

A.Khan
A.Khan

Reputation: 3992

if 1000 records are returned, 1000 lambda functions need to be executed in parallel

What you are trying to achieve is not supported by Step Functions. A State Machine task cannot be modified based on the input it received. So for instance, a Parallel task cannot be configured to add/remove functions based on the number of items it received in an array input.

You should probably consider using SQS Lambda trigger. Number of records retrieved from DB can be added to SQS queue which will then trigger a Lambda function for each item received.

If 30 out of 1000 fail, the main task that was retrieving the records needs to know about it.

There are various ways to achieve this. SQS won't delete an item from the queue if Lambda returns an error. You can configure DLQ and RedrivePolicy based on your requirements. Or you may want to come up with a custom solution to keep the count on failing Lambdas to invoke the service that fetch records from the DB.

Upvotes: 2

Related Questions