tippingpints
tippingpints

Reputation: 59

New AWS EC2 EBS gp3 volumes are slow

Running sysbench against volumes I'm finding the gp3 volumes are much slower. Even when I provision a volume with 16000IOPS after waiting for optimization I'm getting a cap at 1000 IOPS when monitoring through Percona PMM, New Relic CloudWatch.

Instance:

Volumes:

Sysbench results below:

sysbench --file-total-size=15G --file-num=16 fileio prepare
sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --time=600 fileio run
sysbench --file-total-size=15G --file-num=16 fileio cleanup

gp3 | 9000 | 3000 | 250/s

File operations:
reads/s: 576.37
writes/s: 384.24
fsyncs/s: 153.70

Throughput:
read, MiB/s: 9.01
written, MiB/s: 6.00

General statistics:
total time: 600.0333s
total number of events: 668612

Latency (ms):
min: 0.00
avg: 0.90
max: 337.40
95th percentile: 3.89
sum: 599693.33

Threads fairness:
events (avg/stddev): 668612.0000/0.00
execution time (avg/stddev): 599.6933/0.00

gp2 | 9000 | 3000 | gp2

File operations:
reads/s: 1523.68
writes/s: 1015.79
fsyncs/s: 406.33

Throughput:
read, MiB/s: 23.81
written, MiB/s: 15.87

General statistics:
total time: 600.0064s
total number of events: 1767487

Latency (ms):
min: 0.00
avg: 0.34
max: 70.10
95th percentile: 1.06
sum: 599390.12

Threads fairness:
events (avg/stddev): 1767487.0000/0.00
execution time (avg/stddev): 599.3901/0.00

Percona PMM gp3 > gp2 comparison:

Percona PMM gp3 > gp2 comparison

Upvotes: 5

Views: 5991

Answers (1)

Matt L
Matt L

Reputation: 39

My initial enthusiasm for gp3 was dampened by inferior performance to gp2, however, when I set the size, IOPS, and bandwidth the same on C5, M5, and M5A instance type I got similar performance measurements from fio: https://github.com/axboe/fio

I'm testing on CentOS 7.8, fio-3.7-2.el7.x86_64. Region us-east-1.

Are you sure sysbench is the right benchmarking tool? The fileio options don't seem greatly documented, and I can get wildly different results when changing the sync mode, flags, etc.

Your example command equivalent to this because of the default values:

sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --file-io-mode=sync --file-rw-ratio=1.5 --file-fsync-freq=100 --max-requests=0 --time=600 fileio run

When I run a different sysbench command, notably using direct access and fsync=off:

sysbench fileio --file-total-size=15G --file-test-mode=rndrw --time=600 --max-requests=0 --file-io-mode=async --file-extra-flags=direct  --file-num=16 --file-rw-ratio=1 --file-fsync-freq=0 run

I get equivalent performance from gp2 and gp3. (Actually, gp3 is slightly better).

Here's a summary of my fio data, using the command

fio --rw=TEST --direct=1 --ioengine=libaio --bs=16k --numjobs=8 --size=1G --group_reporting

with TEST={read,write,randread,randwrite,randrw}

gp2:

gp2-randread-16k.out:   read:   IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170117msec)
gp2-randrw-16k.out:     read:   IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170121msec)
gp2-randrw-16k.out:     write:  IOPS=1541,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170121msec)
gp2-randwrite-16k.out:  write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170116msec)
gp2-read-16k.out:       read:   IOPS=3090,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169642msec)
gp2-write-16k.out:      write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170115msec)

gp3:

gp3-randread-16k.out:   read:   IOPS=3080,  BW=48.1MiB/s  (50.5MB/s)(8192MiB/170188msec)
gp3-randrw-16k.out:     read:   IOPS=1539,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170238msec)
gp3-randrw-16k.out:     write:  IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170238msec)
gp3-randwrite-16k.out:  write:  IOPS=3082,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170073msec)
gp3-read-16k.out:       read:   IOPS=3089,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169719msec)
gp3-write-16k.out:      write:  IOPS=3083,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170053msec)

Upvotes: 2

Related Questions