Silly John
Silly John

Reputation: 1704

SQS FIFO Queue Integration with Lambda - Ordering Issue during Errors

We are having a NodeJs Lambda which is triggered by an SQS FIFO Queue. The SQS Queue Settings are as follows:

    "sourceQueue": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": "dev-lambdaboilerplate-sourcequeue.fifo",
        "VisibilityTimeout": 30,
        "MessageRetentionPeriod": 1209600,
        "FifoQueue": true,
        "RedrivePolicy": {
          "deadLetterTargetArn": {
            "Fn::GetAtt": [
              "sourceQueueDLQ",
              "Arn"
            ]
          },
          "maxReceiveCount": 4
        }
      }
    },
    "sourceQueueDLQ": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": "dev-lambdaboilerplate-sourcequeue-DLQ.fifo",
        "MessageRetentionPeriod": 1209600,
        "FifoQueue": true
      }
    }
  }

The Lambda Trigger setting is as follows:

"SqspollerEventSourceMappingSQSSourceQueue": {
  "Type": "AWS::Lambda::EventSourceMapping",
  "DependsOn": [
    "IamRoleLambdaExecution"
  ],
  "Properties": {
    "BatchSize": 10,
    "EventSourceArn": {
      "Fn::GetAtt": [
        "sourceQueue",
        "Arn"
      ]
    },
    "FunctionName": {
      "Fn::GetAtt": [
        "SqspollerLambdaFunction",
        "Arn"
      ]
    },
    "Enabled": true,
    "FunctionResponseTypes": [
      "ReportBatchItemFailures"
    ],
    "ScalingConfig": {
      "MaximumConcurrency": 10
    }
  }
}

We are getting two messages into SQS in the below mentioned order:

  1. First Message - The details are as below:
"body": "First Message",
"attributes": {
    "ApproximateReceiveCount": "1",
    "SentTimestamp": "1729084272552",
    "SequenceNumber": "18889389647482863616",
    "MessageGroupId": "groupid1",
    "SenderId": "testid",
    "MessageDeduplicationId": "dedupid1",
    "ApproximateFirstReceiveTimestamp": "1729084272552"
},
  1. Second Message - The details are as below:
"body": "Second Message",
"attributes": {
    "ApproximateReceiveCount": "1",
    "SentTimestamp": "1729084273420",
    "SequenceNumber": "18889389647705071616",
    "MessageGroupId": "groupid1",
    "SenderId": "testid",
    "MessageDeduplicationId": "dedupid2",
    "ApproximateFirstReceiveTimestamp": "1729084302591"
},

Both the messages are having the same group id and ideally should be processed in the same order. These two messages are picked in the below order by the Lambda:

  1. First Lambda Trigger - First Message gets picked (and it errors out and the message is placed back in the queue)
  2. Second Lambda Trigger - First and Second Message gets picked together. The First Message gets errored out and the Second message gets succeeded. Since the First Message errored out, it goes back to the queue
  3. Third Lambda Trigger - First Message gets picked and it gets processed.

This is resulting in First Message being processed after the Second Message. Is there any way, the messages to be ordered sequentially? As per AWS documentation, the second message should not be picked until the first message is processed. Any idea what is going wrong here?

Upvotes: 0

Views: 32

Answers (0)

Related Questions