CODEWITHSUNDEEP
amazon-web-servicesaws-step-functions
vimala
vimala

Reputation: 53

How to Auto scale workers in AWS step functions

We are implementing a process using AWS step functions. Some of the tasks in this process take long time. We are using Activities(workers) running in ECS instances for this step. The state machine is like this:

Trigger -> step 1 (Lambda) -> step2 (ECS) -> final step

Is there a way to find the number of tasks in queue waiting to be picked up by the step2 workers and use this information to auto scale the workers (ECS cluster)?

Upvotes: 5

Views: 2584

Answers (2)

Naor Bar
Naor Bar

Reputation: 2209

I found here a very nice use case and for Step Function autoscaling.

I'm going to use some of those insights, such as using cloudwatch alarms to set get the current state (i.e. scale-up /scale-down decisions); using SQS to hold the alarms etc.

Upvotes: 0

itai ariel
itai ariel

Reputation: 569

You can call the api list_executions to get the list of running executions and than for each of this executions call get_execution_history.

In the response you can iterate the execution events. If you find in the list ActivityScheduled and you don't see ActivityStarted it means it is waiting for a worker to take this task.

Upvotes: 3

Related Questions