CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqs
Memphis Meng
Memphis Meng

Reputation: 1671

Understanding SQS message receive amount

I have a queue which is supposed to receive the messages sent by a lambda function. This function is supposed to send each different message once only. However, I saw a scary amount of receive count on the console: enter image description here

Since I cannot find any explanation about receive count in the plain English, I need to consult StackOverflow Community. I have 2 theories to verify:

  1. There are actually not so many messages and the reason why "receive count" is that high is simply because I polled the messages for a looooong time so the messages were captured more than once;
  2. the function that sends the messages to the queue is SQS-triggered, those messages might be processed by multiple processors. Though I set VisibilityTimeout already, are the messages which are processed going to be deleted? If they aren't remained, there are no reasons for them to be caught and processed for a second time.

Any debugging suggestion will be appreciated!!

Upvotes: 10

Views: 17121

Answers (2)

AnthoB
AnthoB

Reputation: 21

I came accros this post a long time after it has been published. However, looking at what you describe and regarding the delay between the messages, what comes to my mind is : did you set a reserved concurrency on the consumer Lambda function ?

Indeed, having a Lambda triggered for each and every message posted to the queue can make this lambda raise the maximum concurrency assigned, and would cause the messages while the lambda is still running to be continuously consumed / added back to the queue, as the event trigger will not be able to start a new Lambda instance to process it.

A way to overcome it is to set a batch size + maximum batching window to the Lambda SQS trigger, so that messages are "batched" before being sent once to a consumer Lambda. Also, AWS introduced last year the "Maximum Concurrency" parameter for the Lambda SQS event, which permits to avoid this phenomenon.

https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/

Upvotes: 1

Timo Reymann
Timo Reymann

Reputation: 885

So, receive count is basically the amount of times the lambda (or any other consumer) has received the message. It can be that a consumer receives a message more than once (this is by design, and you should handle that in your logic).

That being said, the receive count also increases if your lambda fails to process the message (or even hits the execution limits). The default is 3 times, so if something with your lambda is wrong, you will have at least 3 receives per message.

Also, when you are polling the message, via the AWS console, you are basically increasing the receive count.

Upvotes: 16

Related Questions