CODEWITHSUNDEEP
amazon-dynamodbaws-lambdascalabilityamazon-dynamodb-streams
user1569339
user1569339

Reputation: 683

Increasing DynamoDB Stream + Lambda throughput

I have a DynamoDB Stream that triggers a Lambda function. I'm noticing that bursts of a thousand of writes to the DynamoDB table can take many minutes (longest I have seen is 30 minutes) to all be processed by Lambda. The average duration for each Lambda invocation with batch size of 3 is around 2 seconds. These Lambdas perform I/O heavy tasks, so a small batch size and a higher number of parallel invocations is advantageous. However, the parallelism of these Lambdas is pegged to the number of DynamoDB Stream shards, but I cannot find a method for scaling the number of shards.

Is there any way to increase the throughput of these Lambdas beyond using a bigger batch size and more optimized code?

Upvotes: 9

Views: 8055

Answers (3)

user19279352
user19279352

Reputation: 1

For the DynamoDB Trigger, you could try to increase the Batch Size to process records at once as much as possible to reduce the time cost.

Upvotes: 0

Alexander Patrikalakis
Alexander Patrikalakis

Reputation: 5205

Each stream shard is associated with a partition in DynamoDB. If you increase the throughput on your table so much that you cause your partitions to split, then, you will get more shards. With more shards the number of Lambda functions that run in parallel will increase.

Upvotes: 4

Udo Held
Udo Held

Reputation: 12538

I don't see much configuration options either.

You could decouple your processing. If your change records aren't too large your incoming Lambda could just split them up into several smaller SNS messages. Each of those smaller SNS messages could than trigger a Lambda doing the actual processing. If the changes are larger you could use SQS or S3 and trigger Lambda processing for new messages via SNS or directly for files.

Upvotes: 6

Related Questions