CODEWITHSUNDEEP
c#amazon-web-serviceslambdaasp.net-coreaws-lambda
Andrei Tarutin
Andrei Tarutin

Reputation: 683

Scaling AWS Lambda

I have a queue with a lot go messages coming from different sources. I also have lambda to process the messages in this queue. The point is I can read only up to 10 messages per one SQS request. Since we have time limit for lambda it means I can make not more than 10 invocations in my case per one lambda function run. How can I "tell" lambda to run multiple function instances at a time to process more messages per time from the queue?

Upvotes: 0

Views: 480

Answers (4)

Lee
Lee

Reputation: 841

To answer your question directly, a solution may be to change your lambda to do the following:

  1. Receive the messages from SQS
  2. If more than zero were received, use the Lambda API to self-invoke. This will cause a parallel execution of another instance of the Lambda to run while the current invocation is processing the messages in step 3. You must use the Event invocation type for this to work..
  3. Process the messages.

This is a form of recursion with the base case being "no more messages in the queue". Each invocation of your lambda will process one batch, with batches being processed in parallel so long as more messages can be pulled from the queue. Once the queue is empty, no more invocations will be created until the next time the lambda is triggered by the timer.

This isn't a great solution, but it will work effectively for you and shouldn't be a significant change to implement since it doesn't require you to change architecture at all.

[1] https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html

Upvotes: 0

asdf
asdf

Reputation: 3067

Unfortunately, given your current setup there's not really a good way to optimize it without significant overhead. If you want more than one lambda to be triggered by your timer, you could have multiple timers or a faster timer. However, you would need a large amount of overhead to ensure it is working as expected. There are a couple of solutions you could consider to make this more efficient:

  1. Use Kinesis Streams
  2. Send messages through SNS instead of buffering in a SQS Queue and process them real-time
  3. Move lambda code into a docker container and create an AWS service out of it that can act as a multiprocessor for SQS (i.e. have 5 instances of the container running at a time with scale-up/down triggers)

I would highly recommend the SNS option if you can avoid using SQS since it has the least overhead. Under the hood of every AWS integration method for a lambda lies an SNS endpoint anyway. It's how the service is set up. SNS integration will bring up an instance of the lambda every time a message is sent and scale with your usage instead of you needing to manage your scaling and distribution. That will cause unnecessary headaches.

As for running a lambda on a timer, it is bad practice to use a service that is set up to be event driven on a timer. Personally, when I develop for AWS, all of my event driven functionality is Lambda and all of my time driven functionality is written into docker containers then run on ECS. Running lambda on a timer is something that was developed after the fact due to customers requesting it, not something the service was intitially intended for.

Upvotes: 1

Kannaiyan
Kannaiyan

Reputation: 13055

I would recommend to use Kinesis where all these are automated.

You can set the batch size to your desired size and it will get delivered in batches. If you want to run them in parallel, you can either create multiple streams and assign it to the same Lambda, it will create instance for every stream.

Upvotes: 0

Deividi Silva
Deividi Silva

Reputation: 846

I think the best fit for you would be using Kinesis streams instead of SQS.

AWS documentation on concurrent Lambda execution

Upvotes: 1

Related Questions