CODEWITHSUNDEEP
amazon-web-servicesaws-lambda
user928112
user928112

Reputation: 582

Why does AWS Lambda run in "bursts" every 15 minutes?

First, here's a brief description of how my Lambda function works:

  1. Triggered by messages arriving in an SQS standard queue
  2. Reserved concurrency set to 4
  3. Each SQS message contains a URLs to a list of images, which the Lambda function transfers to an S3 bucket

So in a nutshell: SQS -> Lambda -> S3

In this test: the SQS queue was given 100 messages, each containing an average of 5 image URIs. Each image has an average size of 10 MB.

What I expected was for the Lambda instances to run close together continuously until the SQS queue was empty. Instead, what I saw is this:

enter image description here

It looks like the Lambda function runs in "bursts" once every 15 minutes. My first guess is that this is something I set in my configuration when first deploying the Lambda, but unfortunately I don't quite remember what those initial settings were. I also don't seem to be able to change anything now except for throttling and timeout settings.

Any idea what's going on here?

Upvotes: 0

Views: 1217

Answers (2)

user928112
user928112

Reputation: 582

I figured out why this is happening, but unfortunately there's no solution other than to increase the reserved concurrency.

This has to do with SQS triggered Lambdas that run at a lowered reserved concurrency setting. Long story short, each time SQS attempts to deliver messages to the Lambda function, one or more batch deliveries might fail due to lack of available Lambda instances, and these messages end up getting stuck in flight where they stay until the SQS visibility timeout is reached (which in my case is 15 minutes). At this point the cycle will repeat. If you've configured a DLQ for your SQS, it's possible that you will see some messages end up in there once they've failed enough times.

See this article for a more thorough explanation: https://medium.com/@zaccharles/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0

Upvotes: 2

John Rotenstein
John Rotenstein

Reputation: 270089

From Managing concurrency for a Lambda function - AWS Lambda:

When a function has reserved concurrency, no other function can use that concurrency. Reserved concurrency also limits the maximum concurrency for the function, and applies to the function as a whole, including versions and aliases.

Since you set Reserved Concurrency to 4, only 4 functions will run in parallel.

Thus, it looks like Lambda is triggered 4 functions, waiting for a function to finish, then running the next function. I assume that your function takes about 10-15 seconds to run, resulting in the behavior you see.

Unless you have a particular reason for needing Reserved Concurrency, I suggest that you remove that setting.

Upvotes: 2

Related Questions