Reputation: 534

How does AWS Lambda determine if messages are still in SQS queue?

When using AWS Lambda with a SQS queue (as event source), it is written in the doc

If messages are still available, Lambda increases the number of processes that are reading batches by up to 60 more instances per minute. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

My question here is how does the Lambda service determine "If messages are still available" ?

Upvotes: 0

Answers (5)

fedonev

Reputation: 25639

Answering the "how" question in a slightly different way:

Behind the scenes, Lambda operates a "State Manager" control-plane service that discovers work from the queue. State Manager also manages scaling of the fleet of "Poller" workers that do the actual retrieving, batching, invoking, and deleting.

These implementation details are from the Event Source Mapping section of the re:Invent 2022 video A closer look at AWS Lambda (SVS404-R). Here is a screenshot:

Upvotes: 3

Luke

Reputation: 1419

I believe the documentation refers to Lambda polling the queue to know whether there are still messages. Read more about it here.

Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages. Lambda reads messages in batches and invokes your function once for each batch. When your function successfully processes a batch, Lambda deletes its messages from the queue.

Event Source Mapping:

Lambda only sees messages that are visible, via the visibility timeout setting on the SQS queue. This is to prevent other queue consumers processing the message. I believe as an event-source, Lambda receives messages from the SQS queue, via being mapped to it.

Upvotes: 1

Mark B

Reputation: 200411

I imagine it uses the ApproximateNumberOfMessagesVisible metric on the SQS queue to check how many messages are available, and uses that number, plus your batch size configuration, to determine how many more Lambda instances your function needs to be scaled out to.

Upvotes: 1

stdunbar

Reputation: 17435

One of the calls to the SQS API is to get queue attributes (Java API, others similar). This returns a response and one of the attributes of the response is "approximate number of messages". With this you or AWS can determine about how many messages are in the queue.

From this, AWS can determine if it's worth spinning up additional instances. You too can get this information from the queue.

Upvotes: 1

Ayush Bansal

Reputation: 44

As per the documentation you shared,for standard queues, Long Polling is in effect. Long polling basically waits for a certain amount of time to verify if there is a message in the queue. refer to the following docs :

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html

Upvotes: 0

How does AWS Lambda determine if messages are still in SQS queue?

Answers (5)

Related Questions