CODEWITHSUNDEEP
performancesystem-administrationiostat
duduamar
duduamar

Reputation: 3924

Analyzing iostat output

I'm suffering performance issues with my application, and one of my suspects is excessive IO. iostat shows rate of 10K blocks written per second. How can I tell if this is a lot or not? How can I know the limit of the specific machine and disk?

Edit:Following Elliot's request:

iostat output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

      16.39    0.00    0.52   11.43    0.00   71.66

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
 cciss/c0d0      315.20         0.00     10341.80          0      51709

uptime:

2:08am  up 17 days 17:26,  5 users,  load average: 9.13, 9.32, 8.73

top:

top - 02:10:02 up 17 days, 17:27,  5 users,  load average: 8.89, 9.18, 8.72
Tasks: 202 total,   2 running, 200 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.9%us,  0.7%sy,  0.0%ni, 90.5%id,  2.9%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     96556M total,    15930M used,    80626M free,      221M buffers
Swap:   196615M total,       93M used,   196522M free,     2061M cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                 
20027 root      16   0 10.5g 9.8g  12m S   74 10.4   2407:55 /usr/intel/pkgs/java/1.6.0.31-64/jre//bin/java -server -X

Thanks

Upvotes: 2

Views: 4734

Answers (2)

Kevin J. Rice
Kevin J. Rice

Reputation: 3373

Perspective:

  • If it's spinning disk, that's a high value.
  • If it's an SSD or a SAN with a write cache, that's reasonable.

Use iostat -x for wide and extended metrics:

[xxx@xxxxxxxxxx]$ iostat -x 
Linux 2.6.32-358.6.2.el6.x86_64 (boxname.goes.here)     12/12/2013  _x86_64_    (24 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.57    0.00    0.21    0.00    0.00   99.21

Device:  rrqm/s wrqm/s r/s  w/s  rsec/s wsec/s avgrq-sz avgqu-sz await svctm  %util
sda        0.06  28.38 0.04 3.99   0.99 259.15    64.58     0.01  2.11  0.55   0.22

The %util is your friend. If you look at iostat.c (see it at: http://code.google.com/p/tester-higkoo/source/browse/trunk/Tools/iostat/iostat.c) you can see it calculates this percentage by looking at the amount of time (in processor ticks) spent doing IO versus the total number of ticks that have passed. In other words, the PERCENTAGE-UTIL is the percent of time the IO was in a busy state.

Upvotes: 2

Elliot B.
Elliot B.

Reputation: 17661

I can tell you from experience that's a very high block write rate for most systems. However, your system could be perfectly capable of handling that--depends on what kind of hardware you have. What's important is your server load figure and the iowait percentage. If your server load is high (i.e., higher than the number of cores on your system) and your load largely consists of iowait, then you have a problem.

Could you share with us the full output of iostat, uptime, and a snapshot of top -c output while your application is running?

Upvotes: 2

Related Questions