CODEWITHSUNDEEP
amazon-web-servicesaws-step-functions
ajshort
ajshort

Reputation: 3754

Working around AWS Step Function Map concurrency limit

I have a Map task in an AWS Step Function which executes 100-200 lambdas in parallel, each running for a few minutes, then collects the results. However, I'm running into throttling where not all lambdas are started for some time. The relevant AWS documentation says you may experience throttling with more than ~40 items, which is what I believe I'm running into.

Does anyone have any experience with working around this concurrency limitation? Can i have nested Maps, or could I bucket my tasks into several Maps I run in parallel?

Upvotes: 6

Views: 5351

Answers (3)

jameslol
jameslol

Reputation: 1935

AWS now offers a direct solution to this, called "distributed map state", announced at re:invent 2022 https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html

It allows you to perform 10,000 concurrent map state tasks. Under the hood, it runs them as child step function workflows, which can be specified to run as either standard or express workflows

Upvotes: 5

ajshort
ajshort

Reputation: 3754

I ended up working around this 40 item limit by creating 10 copies of the Map task in a Parallel, and bucketing up the task information to split tasks between these 10 copies. This means I can now run ~400 tasks before running into throttling issues. My state machine looks something like this:

state machine

Upvotes: 4

Pooya Paridel
Pooya Paridel

Reputation: 1391

Use nested state machine inside your map state, so you can have ~40 child state machines executing in parallel. Then inside each child state machine use a map state to process ~40 items in parallel. This way you can reach processing ~1600 items in parallel.

But before reaching that you will reach AWS Step Functions Quotas:

https://docs.aws.amazon.com/step-functions/latest/dg/limits.html

Upvotes: 8

Related Questions