CODEWITHSUNDEEP
amazon-dynamodb
Jonathan
Jonathan

Reputation: 11375

Dynamodb Autoscaling not working fast enough

I'm running a simple api that gets an item from a dynamodb table on each call, I have auto scaling set to a minimum of 25 and a maximum of 10 000.

However if I send 15 000 requests with a tool like wrk or hey, I get about 1000 502s,

Why isn't the autoscaling working better? It only scales upto 99RCUs but my max is 10, 000.

Upvotes: 13

Views: 9067

Answers (2)

BenV
BenV

Reputation: 1206

We ran into the same problem when testing DynamoDB autoscaling for short amounts of time, and it turns out the problem is that the scaling events only happen after 5 minutes of elevated throughput (you can see this by inspecting the CloudWatch alarms the autoscaling sets up)

This excellent blog post helped us solve this by creating a Lambda that responds to the CloudWatch API events and improves the responsiveness of the alarms to one minute: https://hackernoon.com/the-problems-with-dynamodb-auto-scaling-and-how-it-might-be-improved-a92029c8c10b

Upvotes: 11

Eyal Ch
Eyal Ch

Reputation: 10066

from: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html

What you defined as "target utilization"?

Target utilization is the ratio of consumed capacity units to provisioned capacity units, expressed as a percentage. Application Auto Scaling uses its target tracking algorithm to ensure that the provisioned read capacity of ProductCatalog is adjusted as required so that utilization remains at or near 70 percent.

also, i think that the main reason that autoscale not works for you, is because your work might not stay elevated for a long time:

"DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes"

DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes. The Application Auto Scaling target tracking algorithm seeks to keep the target utilization at or near your chosen value over the long term. Sudden, short-duration spikes of activity are accommodated by the table's built-in burst capacity. For more information, see Use Burst Capacity Sparingly.

Upvotes: 4

Related Questions