CODEWITHSUNDEEP
throttlingaws-step-functions
user3002273
user3002273

Reputation:

Placing custom execution rate limits on AWS Step Functions

I have a Step functions setup that is spawning a Lambda function. The Lambda functions are getting spawned too fast and causing other services to throttle, so I would like Step functions to have a rate limit on the number of job it kicks off at a given time.

How do I best approach this?

Upvotes: 6

Views: 6015

Answers (3)

Shridha Jalihal
Shridha Jalihal

Reputation: 85

I had the same issue a year ago. I implemented a fargate acting as a rate limiter in front of the stepfunctions and check the number of executions every 60 secs. If it's more than the threshold, I wait or launch a new stepfunction execution. The ratelimiter was launched by a lambda with concurrency limit 1

Upvotes: 0

Wesley Cheek
Wesley Cheek

Reputation: 1726

I've struggled with this throttling issue trying to find the best way to have a lever in front of my database so it doesn't get overwhelmed. Step Functions itself doesn't have any throttling or concurrency settings unfortunately, so the best way is to integrate it with SQS. You can put the SQS queue either in front the Step Functions workflow or in front of the resource you need to protect - I've found the former to be best.

An example:

Files added to an S3 bucket need to be processed and added into a database through a Step Functions workflow. The best way is to add your S3 Events directly into the SQS queue and then start your Step Function workflow from these events. This way, the workflow can be treated as a single processing unit that is throttled before it runs. Step Functions can be linked to the SQS queue either through a lambda function that starts the Step Functions execution, or, preferably, using the (fairly new) EventBridge Pipes.

The overall architecture:

event -> SQS Queue -> Step Functions Workflow -> protected resource

Upvotes: 1

Marcin Sucharski
Marcin Sucharski

Reputation: 1231

I would try setting this limit at Lambda Function side using concurrent execution limit - https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html. This way you can limit maximum number of executions of that specific Lambda Function thus leaving unreserved concurrency pool for rest of functions.

Upvotes: 1

Related Questions