CODEWITHSUNDEEP
node.jsamazon-web-servicesaws-lambdaamazon-sqs
Mikko Viitala
Mikko Viitala

Reputation: 8404

AWS SQS messages does not become available again after visibility timeout

This is most likely something really simple but for some reason my SQS messages does not become available again after visibility timeout. At least this is what I figured since consuming lambda has no log entries indicating that any retries have been triggered.

My use case is that another lambda is feeding SQS queue with JSON entities that then needs to be sent forward. Sometimes there's so much data to be sent that receiving end is responding with HTTP 429.

My sending lambda (JSON body over HTTPS) is deleting the messages from queue only when service is responding with HTTP 200, otherwise I do nothing with the receiptHandle which I think should then keep the message in the queue.

Anyway, when request is rejected by the service, the message does not become available anymore and so it's never tried to send again and is forever lost.

Incoming SQS has been setup as follows:

Associated DLQ has Maximum receives of 100

SQS

The consuming lambda is configured as

lambda

And the actual logic I have in my lambda is quite simple, really. Event it receives is the Records in the queue. Lambda might get more than one record at a time but all records are handled separately.

console.log('Response', response);

if (response.status === 'CREATED') {

  /* some code here */

  const deleteParams = {
    QueueUrl: queueUrl, /* required */
    ReceiptHandle: receiptHandle /* required */
  };

  console.log('Removing record from ', queueUrl, 'with params', deleteParams);    
  await sqs.deleteMessage(deleteParams).promise();
} else {
  /* any record that ends up here, are never seen again :( */
  console.log('Keeping record in', eventSourceARN);
}

What do :( ?!?!11

Upvotes: 1

Views: 2189

Answers (3)

Mikko Viitala
Mikko Viitala

Reputation: 8404

Thanks to everyone who answered. So I got this "problem" solved just by going through the documents.

I'll provide more detailed answer to my own question here in case someone besides me didn't get it on the first go :)

So function should return batchItemFailures object containing message ids of failures.

So, for example, one can have Lambda handler as

/**
 * Handler
 *
 * @param {*} event SQS event
 * @returns {Object} batch item failures
 */
exports.handler = async (event) => {
  console.log('Starting ' + process.env.AWS_LAMBDA_FUNCTION_NAME);
  console.log('Received event', event);

  event = typeof event === 'object'
    ? event
    : JSON.parse(event);

  const batchItemFailures = await execute(event.Records);

  if (batchItemFailures.length > 0) {
    console.log('Failures', batchItemFailures);
  } else {
    console.log('No failures');
  }

  return {
    batchItemFailures: batchItemFailures
  }
}

and execute function, that handles the messages

/**
 * Execute
 *
 * @param {Array} records SQS records
 * @returns {Promise<*[]>} batch item failures
 */
async function execute (records) {
  let batchItemFailures = [];

  for (let index = 0; index < records.length; index++) {
    const record = records[index];

    // ...some async stuff here

    if (someSuccessCondition) {
      console.log('Life is good');
    } else {
      batchItemFailures.push({
        itemIdentifier: record.messageId
      });
    }
  }

  return batchItemFailures;
}

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 269330

When an AWS Lambda function is triggered from an Amazon SQS queue, all activities related to SQS are handled by the Lambda service. Your code should not call any Amazon SQS functions.

The messages will be provided to the AWS Lambda function via the event parameter. When the function successfully exits, the Lambda service will delete the messages from the queue.

Your code should not be calling DeleteMessage().

If you wish to signal that some of the messages were not successfully processed, you can use a partial batch response to indicate which messages were successfully processed. The AWS Lambda service will then make the unsuccessful messages available on the queue again.

Upvotes: 1

Parsifal
Parsifal

Reputation: 4486

otherwise I do nothing with the receiptHandle which I think should then keep the message in the queue

That's not now it works:

Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages. Lambda reads messages in batches and invokes your function once for each batch. When your function successfully processes a batch, Lambda deletes its messages from the queue.

Upvotes: 3

Related Questions