CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqs
Rick
Rick

Reputation: 698

Not all SQS messages end up in Lambda: most just disappear

I have an AWS SQS Queue (standard, non-FIFO) that has a Lambda function as a consumer.

Whenever I send a bunch of messages (usually around 10 at a time) to the queue, only about 2 get picked up by lambda (verified in CloudWatch Logs). The others disappear from the queue.

The Lambda batch size is set to 1, so I would expect all 10 messages to sit in the queue and get picked up by Lambda one by one, but that's not happening. I'm using CloudWatch to check what Lambda is doing, and there is no trace of the missing messages.

I verified in Lambda that it only gets one message every time, by logging the size of the event.Records array (which is always 1).

The Queue also has a Dead Letter Queue. Initially the Maximum Receives was set to 1. When I increased that to 3, more messages were getting picked up after the queues Visibility timeout, but still only a few.

My Queue settings

I'm wondering why the messages aren't being processed, but instead disappear?

Upvotes: 5

Views: 5881

Answers (2)

Rick
Rick

Reputation: 698

Turns out this was related to the Reserved Concurrency of the Lambda function. My concurrency was set to 1, which caused issues.

My expectation SQS messages will remain in the queue until there's a Lambda function available to pick them up.

In reality Messages that are not picked up by Lambda because the function is throttled, and after the Visibility Timeout are treated as failed message.

There's an excellent blog post about this issue: https://data.solita.fi/lessons-learned-from-combining-sqs-and-lambda-in-a-data-project/

Upvotes: 9

John Rotenstein
John Rotenstein

Reputation: 270089

The typical reason why messages are lost is that the Lambda function triggered by Amazon SQS is not correctly processing all Records passed to the function.

Make sure the code loops through all Records passed in the even parameter, since multiple messages can be provided in each Lambda invocation.

Upvotes: 4

Related Questions