CODEWITHSUNDEEP
postgresqlamazon-web-servicesamazon-ec2
pulkit
pulkit

Reputation: 136

AWS-EBS performance degrades after some time

We are using Postgresql as server on EBS with 1TB volume and the data write operations performance very well till 0.7M(6-7 lakh queries) operations after that the the write speed start degrading.

Queries which are taking 0.02 sec to complete starts taking 10-12 seconds.

Disclaimer: We are having a write heavy database with 26 tables which performs write operation in 26 different tables.

The problemis that, CPU usage in our case won't exceed 40% and RAM always have 1.5GB free memory.

We had performed following experiments:

  1. Used 300GB gp2 volume with and without EBS-optimized instance.
  2. Used 300GB volume with PIOPS(with 15000 IOPS) and without EBS-optimized instance.
  3. Used 1TB gp2 volume with and without EBS optimizeed instance.
  4. Used 1TB volume with EBS optimized instance and io1(PIOPS of 15000)

Experiment performed on r3.large, 4 core, 30.5GB RAM for EBS-optimized instances and t2.medium, 2 core, 4GB RAM with non-EBS optimized.

Is this a problem of Postgres or EBS?

Upvotes: 3

Views: 1975

Answers (1)

chris
chris

Reputation: 37460

So your problem appears to be that after a certain amount of time, your write performance slows down.

There can be a couple of causes for this.

First, you will see this type of behavior when using T2 family instances - they are burstable, but taking advantage of additional performance available to T2s only lasts until you run out of credits - then the instance reverts to its default performance, and in practice that makes the instance almost unusable. You can monitor T2 credit balance and credit use from either the EC2 monitor screen or CloudWatch. That can help determine if credit exhaustion is contributing to the problem.

The other cause for this may be burstable EBS performance. The general purpose SSD EBS volumes (gp2) supports bursts of up to 3000 IOPS. Starting in Nov. 2016, AWS has exposed this metric via Cloudwatch. So if you are doing a large amount of IO (which you would expect during a load test) you may be running into has used up its burst balance.

Once you determine the cause of your slowdowns (and it may be a combination of issues) you can determine the best way to resolve it. A simple solution would be to use provisioned iops (io1) volumes for your testing.

Upvotes: 1

Related Questions