CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqs
Amiga500
Amiga500

Reputation: 6141

AWS - SQS Batch Size and Lambda approach

If I understand correctly, batch size setting on Lambda decides how many messages to take in one sweep from the SQS. Therefore this JSON (taken from the test Lambda SQS);

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "FAIL",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {
      },
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:eu-west-1:123456789012:MyQueue",
      "awsRegion": "eu-west-1"
    }
  ]
}

There is Records array. And if I set batch size to 5, then if there are 5 messages in SQS, they will be included in array. If there are 10 messages, then Lambda will be invoked twice, with 5 messages in each Record.

I am now confused a bit, as to what approach to take. My Lambda is fairly simple. It will be Axios POST request to external service. If it errors out, I will throw an error. I could even use axios-retry, and make retries fairly easy.

Should I use batch in my case? Naively lookin, all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

Contrary, I would have to iterate via all messages and attempt an Axios request. What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Upvotes: 26

Views: 40529

Answers (2)

user11666461
user11666461

Reputation: 1145

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

As per AWS documentation, SQS event source mapping now supports handling of partial failures out of the box. For this to work you have to include ReportBatchItemFailures in your EventSourceMapping configuration. So if third of five messages fail don't throw an exception, instead keep a track of the messageId of the failed message and continue processing 4th and 5th message. Return back

{
  "batchItemFailures": [
    { "itemIdentifier": "id3" }
  ]
}

where id3 is the messageId of the 3rd message as response to SQS. For further details refer the documentation and SO post

Upvotes: 3

A.Khan
A.Khan

Reputation: 4002

all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

I don't think you need batch processing in above case.

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Message(s) won't be deleted from the queue if your Lambda errors, based on the SQS visibility timeout and redrive policy configuration. SQS will trigger Lambda again according to the configuration. If you have configured DLQ then after maxReceiveCount is reached, failed messages will be added to the DLQ and removed from the main queue.

Upvotes: 15

Related Questions