CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-dynamodb
incnnu
incnnu

Reputation: 193

DynamoDB write is too slow

I use Lambda to read from a JSON Api and write in DynamoDB via http request. The JSON Api is very big (has 200k objects) and my function is extremely slow with writing to DynamoDB. I used the regular write function and after 10 min I could only populate 5k rows in my DynamoDB table. I was thinking about using BatchWriteItem but since it can only do 25 puts in one batch, it would still take too much time to write all 200k rows. Is there any better solution?

Upvotes: 2

Views: 4670

Answers (1)

Chris Williams
Chris Williams

Reputation: 35258

This will be because you're being throttled.

For Lambda

There are a maximum number of concurrent invocations of Lambdas that can be running at a time, the default limit is 1000 concurrent requests.

If you have more than 1000 concurrent requests at the same time you will need to reach out to AWS Support to increase this, you will also need to provide a business use case for why it needs to support this.

For DynamoDB

Whether you use batch or single PutItem your DynamoDB table is configured with a number of WCU (Write Credit Units) and RCU (Read Credit Units).

A single write credit unit covers 1 write of an item 1Kb or less (every extra kb is another unit). If you exceed this you will start to be throttled for write requests, if you're using the SDK it may use exponential backoff as well to keep attempting to write.

As a solution for this you should do one of the following:

  • If this is a one time process you can adjust the WCU as a fixed number, then wait 5 minutes for it to increase and then scale down.
  • If this is a natural flow on your app then enable DynamoDB autoscaling to naturally increase and decrease throughout the day

In addition look at your data modelling as this can lead to throttling too.

In extreme cases, throttling can occur if a single partition receives more than 3,000 RCUs or 1,000 WCUs

Upvotes: 3

Related Questions