CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqs
Aditya
Aditya

Reputation: 417

Does a batch of messages from SQS become available for processing again, if the lambda timeouts?

There is an SQS which acts as input trigger for a lambda. The batch size of the records to be fetched from the SQS is 10.

While processing these 10 records, if the lambda times out, will all these 10 records/ messages be available for processing again? Or will the ones that are processed get removed from the SQS?

I tried to look for this in a couple of docs: link1, link2. But couldn't find it.

If the 10 messages do become available for processing, then a possible solution I am thinking to implement is: Keep deleting the messages once they are processed. But even if I delete the messages, will they become available once the lambda times out?

Upvotes: 2

Views: 1634

Answers (2)

IMSoP
IMSoP

Reputation: 97968

The page on Visibility Timeouts which you linked to starts with this sentence:

When a consumer receives and processes a message from a queue, the message remains in the queue.

The Lambda function in your example is the "consumer" of messages from SQS. It receives and processes a batch of messages, but those messages are still in the queue.

There is also a page on Using AWS Lambda with Amazon SQS, which briefly mentions this:

Lambda reads messages in batches and invokes your function once for each batch. When your function successfully processes a batch, Lambda deletes its messages from the queue.

Here, the Lambda Event Source provides part of the "consumer" code, and deletes messages that it knows are processed. But it only has two states: your Lambda function succeeded, so the batch should be deleted; or your Lambda function failed, so the batch should not be deleted.

In order to have some messages "succeed" and some "fail", you would need to tell SQS that. There is no separate verb to "acknowledge" each message as you process it; quoting the Visibility Timeouts page again:

Thus, the consumer must delete the message from the queue after receiving and processing it.

So, if your Lambda function wants to mark individual messages as processed, it will need to call the DeleteMessage API.

Upvotes: 3

Jens
Jens

Reputation: 21540

SQS does re-queue a message to your Lambda in two circumstances:

  1. Your Lambda or the Lambda service return/throw an error.
  2. The visibility timeout has been reached without your Lambda completing.

A timeout would be considered an error by the Lambda service. Therefore, the messages (and its records) would be send to your Lambda again. If that fails too often the message is either dropped or send to a dead letter queue, depending on your configuration.

If you delete a message from the queue before you did process it completely, it would not be re-send but you run the risk of data loss.

If you for example get a message with 10 records, immediately delete that message from SQS, then only process 5 of 10 records and then timeout, the last 5 records will never be processed.

So I would recommend to only delete SQS messages which have been processed completely.

Sounds easy, but can become tricky. For example: what if 9 of 10 records where processed successfully, but one was not. In theory, you should return an error so that the whole message (with its 10 records) can be re-queued. But then you would re-process those 9 successful records again. That might become a problem, depending on your application design. Idempotence is your friend here.

Upvotes: 2

Related Questions