CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqs
Tobi Akerele
Tobi Akerele

Reputation: 1001

Processing AWS SQS messages with separate Lambda at a time

Like the title suggests, I have a scenario that I would like to explore but do not know how to go about it.

I have a lambda function processCSVFile. I also have a SQS queue that at a set time everyday, it gets populated with link of csv files from S3, let's say about 2000 messages. Now I want to process 25 messages at a time once the SQS queue has the messages.

The scenario I am looking for is to process 25 messages concurrently, I want the 25 messages to be processed by 25 lambda invocations separately. I thought I could use SendMessageBatch function in SQS but this only delivers messages to the queue, it does not seem to apply to my use case.

My question is, am I able to perform the action explained above and if it is possible, what documentation or use cases can explain what I am looking for. Also, if this use case is impossible, what do you recommend as an alternative way to do the processing I want done concurrently.

Upvotes: 7

Views: 7025

Answers (2)

Marcin
Marcin

Reputation: 238051

I think that combination of lambda's event source mapping for sqs and setting reserved concurrency to 25 could be the way do go.

The lambda uses long pooling to prepare message batches for concurrent processing by lambda. Thus each invocation of your function could get more than 1 message at a time.

I don't think there is a way to set event source mapping to serve just one message per batch. If you absolute must ensure only one message is processed by lambda, then you process one and disregards others (put them back to queue).

The reserved concurrency of 25 guarantees that you wont be running more than 25 functions in parallel. If you leave it at its default value, you can run up to whatever free concurrency you have in your account.

Edit:

@JohnRotenstein already confirmed that there is a way to set lambda to pass message a time to your function.

Hope this helps.

Upvotes: 3

John Rotenstein
John Rotenstein

Reputation: 269081

To process 25 messages from Amazon SQS with 25 concurrent Lambda functions (1 message per running Lambda function), you would need:

  • A maximum concurrency of 25 configured for the Lambda function (otherwise it might go higher than this when more messages are available)
  • A batch size of 1 configured on the Lambda trigger so that SQS only passes it one message at a time

See:

Upvotes: 10

Related Questions