CODEWITHSUNDEEP
amazon-web-servicesamazon-sqsaws-cdk
Maxim Suponya
Maxim Suponya

Reputation: 1514

aws sqs messages consistently disappear

Got 3 SQS queues and 3 lambdas. Lambda1 sends 50 messages to queue1, lambda2 picks up a message from queue1 and sends to queue2, lambda3 picks up off queue2 and relays to queue3. Queues defined via cdk:

const stageOneQ = new sqs.Queue(this, "StageOne", {
  fifo: true,
  queueName: "StageOne.fifo",
});

const stageTwoQ = new sqs.Queue(this, "StageTwo", {
  fifo: true,
  queueName: "StageTwo.fifo",
});

const stageThreeQ = new sqs.Queue(this, "StageThree", {
  fifo: true,
  queueName: "StageThree.fifo",
});

same with lambdas, e.g.:

const stageOneHandler = new lambda.Function(this, "stageOneHandler", {
  runtime: lambda.Runtime.NODEJS_10_X,
  code: lambda.Code.fromAsset("resources"),
  handler: "stage-one-handler.handler",
  timeout: Duration.seconds(30),
  environment: {
    stageTwoQ: stageTwoQ.queueUrl
  }
});

queues set as lambda2 and 3 event sources, e.g.:

stageOneHandler.addEventSource(new SqsEventSource(stageOneQ));

and messages are found in lambda events:

exports.handler = async function(event, context) {
   console.log(`Received: ${event.Records[0].body}`);
 .....

lambdas send messages like so (profile.email is unique):

await sqs.sendMessage({
  MessageBody: message,
  MessageGroupId: "group1",
  MessageDeduplicationId: profile.email,
  QueueUrl: queueUrl
}).promise();

queue1 receives 50 messages as expected, but queue2 receives between 18 and 26, and queue3 around 6 or 12.. if lambda1 sends the 50 messages with 1 sec interval, all messages are delivered to all queues. When interval is reduced to 100ms or less some messages disappear with no trace, sent but never received. Why?! Is AWS SQS.. bad? Tried content based de-duplication, didn't help.

Upvotes: 1

Views: 929

Answers (1)

Mark B
Mark B

Reputation: 200436

In this code:

console.log(`Received: ${event.Records[0].body}`);

You are only logging the first record in the list. event.Records is an array. You need to iterate over that array and process all the records in it. By default the Lambda SQS integration will send records in batches of up to 10 at a time. You would have to override the Lambda/SQS integration batch size setting with a value of 1 to only get one message at a time.

Upvotes: 3

Related Questions