HDFS stress test in multi-node Hadoop cluster

Question

I have set up a multi-node Hadoop cluster. I'm intended to test the performance of the HDFS without using MR. Is there such a benchmark in Hadoop test jar?

Thanks

Answer 1

TestDFSIO:

1. The default output directory is /benchmarks/TestDFSIO
2. Run write tests before read tests

The syntax for running a write test is as follows:

TestDFSIO.0.0.4
Usage: hadoop jar $HADOOP_HOME/hadoop-*test*.jar TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes]

Eg: The command to run a write test that generates 10 output files of size 1GB for a total of 10GB is:

$ hadoop jar hadoop-*test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000

Similarly, read test using 10 input files of size 1GB is:

$ hadoop jar hadoop-*test*.jar TestDFSIO -read -nrFiles 10 -fileSize 1000

Clean up and remove test data using: $ hadoop jar hadoop-*test*.jar TestDFSIO -clean

Interpreting TestDFSIO results:

Sample:

----- TestDFSIO ----- : write
           Date & time: Fri Apr 08 2011
       Number of files: 1000
Total MBytes processed: 1000000
     Throughput mb/sec: 4.989
Average IO rate mb/sec: 5.185
 IO rate std deviation: 0.960
    Test exec time sec: 1113.53

----- TestDFSIO ----- : read
           Date & time: Fri Apr 08 2011
       Number of files: 1000
Total MBytes processed: 1000000
     Throughput mb/sec: 11.349
Average IO rate mb/sec: 22.341
 IO rate std deviation: 119.231
    Test exec time sec: 544.842

The most notable metrics here are Throughput mb/sec and Average IO rate mb/sec.

Source