CODEWITHSUNDEEP
amazon-web-servicesstate-machineaws-step-functions
Master
Master

Reputation: 103

AWS Step Function: How can i invoke multiple instances of same lambda in step function

I have an SQS Queue of which I monitor its size from a state machine.

If size > desired size then I trigger some lambda functions, otherwise, it waits for 30 seconds and checks the queue size again.

Here is my probem: when the queue length is > 20000 I want to trigger 10 lambda functions to empty it faster. And if its length is <2000 then I want to only run 1 lambda function.

For now, I have hard coded ten parallel steps but its waste of resources if the queue size is less than 2000.

            "CheckSize": {
              "Type": "Choice",
              "Choices": [
                {
                  "Variable": "$.Payload.size",
                  "NumericGreaterThan": 2000,
                  "Next": "invoke_lambda"
                },
                {
                  "Variable": "$.Payload.size",
                  "NumericLessThan": 2000,
                  "Next": "Wait30s"
                }
              ],
              "Default": "Wait30s"
            },

Upvotes: 5

Views: 11200

Answers (4)

swarnim gupta
swarnim gupta

Reputation: 231

AWS Step Functions now supports dynamic parallelism, so you can optimize the performance and efficiency of application workflows such as data processing and task automation. By running identical tasks in parallel, you can achieve consistent execution durations and improve utilization of resources to save on operating costs. Step Functions automatically scales resources in response to your input.

https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html

https://aws.amazon.com/about-aws/whats-new/2019/09/aws-step-functions-adds-support-for-dynamic-parallelism-in-workflows/

Upvotes: 1

Warren Parad
Warren Parad

Reputation: 4073

SQS now supports using Lambda as a EventSourceMapping so the recommendation would be to have AWS directly take control of this and scale lambdas as necessary.

Example CloudFormation template would be

"EventSourceMapping": {
      "Type": "AWS::Lambda::EventSourceMapping",
      "Properties": {
        "BatchSize": 10,
        "Enabled": true,
        "EventSourceArn" : { "Fn::GetAtt" : ["SQSStandupWork", "Arn"] },
        "FunctionName" : {
          "Fn::Join": [
            ":", [
              { "Fn::GetAtt" : ["LambdaFunction", "Arn"] },
              "production"
            ]
          ]
        }
      }
    }

If you are really set on using a step function to drive this forward, you can create another choice on top of what you currently have

  • A - execute in parallel 1 lambda (A1 => stop) + a checker (B)

  • B - call lambda and check the size return Wait30 (B1 if size is less than 2000), return Parallel if size is > 20000 (B2)

  • B1 - wait 30 and then NEXT: A

  • B2 - Have a parallel with 9 lambdas (since the 10 is A) => NEXT: A

Additional alternatives are:

  • CloudWatch event to schedule triggering every 30seconds

  • trigger the 10 parallel lambda functions directly from a separate lambda. A lambda could check the size and then directly call the other lambdas in ASYNC. since it doesn't matter what the result is, since we'll check again in 30 seconds the step function will retry.

The biggest problem with your suggested approach is that the step function has a 1 year limit, so unless you are sure the queue will be drained within a year you'll have a problem when you get to the end. Even if you set it up to retrigger a new step function, you'll be paying a lot of unnecessary step transitions (step functions are not the cheapest.)

Upvotes: 0

A.Khan
A.Khan

Reputation: 3992

AWS Step Functions does not appear to be the best tool in your scenario. I think you should be using one of the SQS metrics available for CloudWatch. It should be ApproximateNumberOfMessagesVisible in your case. You can create an alarm if ApproximateNumberOfMessagesVisible >= 20,000. Action for that alarm would probably be SNS topic to which you can subscribe a Lambda function. In the Lambda function you can asynchronously invoke your Lambda function 10 times that is supposed to clear down the queue.

Check out AWS docs for creating a CloudWatch alarm for SQS metric

Using Step Functions:

If you want to do it with Step Functions then I don't think you need any Condition check in your state machine definition. All you need is to pass the $.size to a Lambda function and put the condition in that Lambda function. If size >= 20000 then asynchronously invoke queue processing function 10 times else 1.

Upvotes: 1

Dhaval
Dhaval

Reputation: 1076

Not diving deep into the solution that you have come up with and focusing on providing guidance on your question

So, If you see, you have answer the question yourself. The simplest solution is to make one more step called invoke10Lambdas and use it from your choice. Pseudo code for your step function would look something like this.

....
....
"CheckSizeAndDivert": {
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.Payload.size",
      "NumericGreaterThan": 20000,
      "Next": "invoke_10_lambdas"
    },
    {
      "Variable": "$.Payload.size",
      "NumericGreaterThan": 2000,
      "Next": "invoke_lambda"
    }
  ],
  "Default": "Wait30s"
},

"invoke_10_lambdas": {
// This is your parallel step.
...
Next:"whatever next(i believe it is Wait30)"
},

"invoke_lambda": {
...
// This is your single lambda step.
...
Next:"whatever next(i believe it is Wait30)"
},
...
...

Upvotes: 1

Related Questions