CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambdamessage-queueamazon-sqs
muazfaiz
muazfaiz

Reputation: 5021

Processing AWS multiple SQS messages with one Lambda Function

I have setup an SQS queue where S3 paths are being pushed whenever there is a file upload.

So I have a setup where I'll receive 10s of small csv files and I want to hold them in a SQS queue and trigger the lambda only once when all the files have arrived during a specific time let's say 5 minutes.

Here is my CF code

  LambdaFunctionEventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 5000
      MaximumBatchingWindowInSeconds: 300
      Enabled: true
      EventSourceArn: !GetAtt EventQueue.Arn
      FunctionName: !GetAtt QueueConsumerLambdaFunction.Arn

  EventQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: Event-Queue
      DelaySeconds: 10
      VisibilityTimeout: 125
      ReceiveMessageWaitTimeSeconds: 10

  QueueConsumerLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: queue-consumer
      Runtime: python3.7
      Code: ./queue-consumer
      Handler: main.lambda_handler
      Role: !GetAtt QueueConsumerLambdaExecutionRole.Arn
      Timeout: 120
      MemorySize: 512
      ReservedConcurrentExecutions: 1

The deployment works fine but if I push 3 files to S3 bucket the SQS triggers 3 different lambda functions asynchronously which I don't want. I need one lambda function to contain all messages in the queue as a result of S3 event and process them. Is there something wrong in my SQS configuration?

Upvotes: 0

Views: 2509

Answers (1)

Marcin
Marcin

Reputation: 238051

What you are observing is likely due to five parallel threads that AWS is using to query your SQS queue. These threads are separate from concurrency setting, and you have no control over these threads. There are always 5 of them.

So each thread will get some msgs from the queue, then your function is going to be invoked with these msgs in turn. Sadly you can't change how it works, as this is how sqs and lambda work at AWS side.

Upvotes: 1

Related Questions