Testing disk performance: differs with and without using Java

Question

I've been asked to measure current disk performance, as we are planning to replace local disk with network attached storage on our application servers. Since our applications which write data are written in Java, I thought I would measure the performance directly in Linux, and also using a simple Java test. However I'm getting significantly different results, particularly for reading data, using what appear to me to be similar tests. Directly in Linux I'm doing:

dd if=/dev/zero of=/data/cache/test bs=1048576 count=8192
dd if=/data/cache/test of=/dev/null bs=1048576 count=8192

My Java test looks like this:

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;

public class TestDiskSpeed {
private byte[] oneMB = new byte[1024 * 1024];

public static void main(String[] args) throws IOException {
    new TestDiskSpeed().execute(args);
}

private void execute(String[] args) throws IOException {
    long size = Long.parseLong(args[1]);
    testWriteSpeed(args[0], size);
    testReadSpeed(args[0], size);
}


private void testWriteSpeed(String filePath, long size) throws IOException {
    File file = new File(filePath);
    BufferedOutputStream writer = null;
    long start = System.currentTimeMillis();
    try {
        writer = new BufferedOutputStream(new FileOutputStream(file), 1024 * 1024);
        for (int i = 0; i < size; i++) {
            writer.write(oneMB);
        }
        writer.flush();
    } finally {
        if (writer != null) {
            writer.close();
        }
    }
    long elapsed = System.currentTimeMillis() - start;
    String message = "Wrote " + size + "MB in " + elapsed + "ms at a speed of " + calculateSpeed(size, elapsed) + "MB/s";
    System.out.println(message);
}

private void testReadSpeed(String filePath, long size) throws IOException {
    File file = new File(filePath);
    BufferedInputStream reader = null;
    long start = System.currentTimeMillis();
    try {
        reader = new BufferedInputStream(new FileInputStream(file), 1024 * 1024);
        for (int i = 0; i < size; i++) {
            reader.read(oneMB);
        }
    } finally {
        if (reader != null) {
            reader.close();
        }
    }
    long elapsed = System.currentTimeMillis() - start;
    String message = "Read " + size + "MB in " + elapsed + "ms at a speed of " + calculateSpeed(size, elapsed) + "MB/s";
    System.out.println(message);
}

private double calculateSpeed(long size, long elapsed) {
    double seconds = ((double) elapsed) / 1000L;
    double speed = ((double) size) / seconds;
    return speed;
}

}

This is being invoked with "java TestDiskSpeed /data/cache/test 8192"

Both of these should be creating 8GB files of zeros, 1MB at a time, measuring the speed, and then reading it back and measuring again. Yet the speeds I'm consistently getting are:

Linux: write - ~650MB/s

Linux: read - ~4.2GB/s

Java: write - ~500MB/s

Java: read - ~1.9GB/s

Can anyone explain the large discrepancy?

Answer 1

To complement Peter's great answer, I am adding the code below. It compares head-to-head the performance of the good-old java.io with NIO. Unlike Peter, instead of just reading data into a direct buffer, I do a typical thing with it: transfer it into an on-heap byte array. This steals surprisingly little from the performance: where I was getting 7.5 GB/s with Peter's code, here I get 6.0 GB/s.

For the java.io approach I can't have a direct buffer, but instead I call the read method directly with my target on-heap byte array. Note that this array is smallish and has an awkward size of 555 bytes. Nevertheless I retrieve almost identical performance: 5.6 GB/s. The difference is so small that it would evaporate completely in normal usage, and even in this artificial scenario if I wasn't reading directly from the disk cache.

As a bonus I include at the bottom a method which can be used on Linux and Mac to purge the disk caches. You'll see a dramatic turn in performance if you decide to call it between the write and the read step.

public final class MeasureIOPerformance {
        static final int SIZE_GB = Integer.getInteger("sizeGB", 8);
        static final int BLOCK_SIZE = 64 * 1024;
        static final int blocks = (int) (((long) SIZE_GB << 30) / BLOCK_SIZE);
        static final byte[] acceptBuffer = new byte[555];

        public static void main(String[] args) throws IOException {
            for (int i = 0; i < 3; i++) {
                measure(new ChannelRw());
                measure(new StreamRw());
            }
        }

        private static void measure(RW rw) throws IOException {
            File file = File.createTempFile("delete", "me");
            file.deleteOnExit();
            System.out.println("Writing " + SIZE_GB + " GB " + " with " + rw);
            long start = System.nanoTime();
            rw.write(file);
            long mid = System.nanoTime();
            System.out.println("Reading " + SIZE_GB + " GB " + " with " + rw);
            long checksum = rw.read(file);
            long end = System.nanoTime();
            long size = file.length();
            System.out.printf("Write speed %.1f GB/s, read Speed %.1f GB/s%n",
                    (double) size/(mid-start), (double) size/(end-mid));
            System.out.println(checksum);
            file.delete();
        }

        interface RW {
            void write(File f) throws IOException;
            long read(File f) throws IOException;
        }

        static class ChannelRw implements RW {
            final ByteBuffer directBuffer = ByteBuffer.allocateDirect(BLOCK_SIZE);

            @Override public String toString() {
                return "Channel";
            }

            @Override public void write(File f) throws IOException {
                FileChannel fc = new FileOutputStream(f).getChannel();
                try {
                    for (int i = 0; i < blocks; i++) {
                        directBuffer.clear();
                        while (directBuffer.remaining() > 0) {
                            fc.write(directBuffer);
                        }
                    }
                } finally {
                    fc.close();
                }
            }
            @Override public long read(File f) throws IOException {
                ByteBuffer buffer = ByteBuffer.allocateDirect(BLOCK_SIZE);
                FileChannel fc = new FileInputStream(f).getChannel();
                long checksum = 0;
                try {
                    for (int i = 0; i < blocks; i++) {
                        buffer.clear();
                        while (buffer.hasRemaining()) {
                            fc.read(buffer);
                        }
                        buffer.flip();
                        while (buffer.hasRemaining()) {
                            buffer.get(acceptBuffer, 0, Math.min(acceptBuffer.length, buffer.remaining()));
                            checksum += acceptBuffer[acceptBuffer[0]];
                        }
                    }
                } finally {
                    fc.close();
                }
                return checksum;
            }
        }

        static class StreamRw implements RW {
            final byte[] buffer = new byte[BLOCK_SIZE];

            @Override public String toString() {
                return "Stream";
            }

            @Override public void write(File f) throws IOException {
                FileOutputStream out = new FileOutputStream(f);
                try {
                    for (int i = 0; i < blocks; i++) {
                        out.write(buffer);
                    }
                } finally {
                    out.close();
                }
            }
            @Override public long read(File f) throws IOException {
                FileInputStream in = new FileInputStream(f);
                long checksum = 0;
                try {
                    for (int i = 0; i < blocks; i++) {
                        for (int remaining = acceptBuffer.length, read;
                             (read = in.read(buffer)) != -1 && (remaining -= read) > 0; )
                        {
                            in.read(acceptBuffer, acceptBuffer.length - remaining, remaining);
                        }
                        checksum += acceptBuffer[acceptBuffer[0]];
                    }
                } finally {
                    in.close();
                }
                return checksum;
            }
        }


        public static void purgeCache() throws IOException, InterruptedException {
            if (System.getProperty("os.name").startsWith("Mac")) {
                new ProcessBuilder("sudo", "purge")
    //                    .inheritIO()
                        .start().waitFor();
            } else {
                new ProcessBuilder("sudo", "su", "-c", "echo 3 > /proc/sys/vm/drop_caches")
    //                    .inheritIO()
                        .start().waitFor();
            }
        }
    }

Answer 2

When I run this using NIO on my system. Ubuntu 15.04 with an i7-3970X

public class Main {
    static final int SIZE_GB = Integer.getInteger("sizeGB", 8);
    static final int BLOCK_SIZE = 64 * 1024;

    public static void main(String[] args) throws IOException {
        ByteBuffer buffer = ByteBuffer.allocateDirect(BLOCK_SIZE);
        File tmp = File.createTempFile("delete", "me");
        tmp.deleteOnExit();
        int blocks = (int) (((long) SIZE_GB << 30) / BLOCK_SIZE);
        long start = System.nanoTime();
        try (FileChannel fc = new FileOutputStream(tmp).getChannel()) {
            for (int i = 0; i < blocks; i++) {
                buffer.clear();
                while (buffer.remaining() > 0)
                    fc.write(buffer);
            }
        }
        long mid = System.nanoTime();
        try (FileChannel fc = new FileInputStream(tmp).getChannel()) {
            for (int i = 0; i < blocks; i++) {
                buffer.clear();
                while (buffer.remaining() > 0)
                    fc.read(buffer);
            }
        }
        long end = System.nanoTime();

        long size = tmp.length();
        System.out.printf("Write speed %.1f GB/s, read Speed %.1f GB/s%n",
                (double) size/(mid-start), (double) size/(end-mid));

    }
}

prints

Write speed 3.8 GB/s, read Speed 6.8 GB/s

Answer 3

You may get better performance if you drop the BufferedXxxStream. It's not helping since you're doing 1Mb read/writes, and is cause extra memory copy of the data.

Better yet, you should be using the NIO classes instead of the regular IO classes.

try-finally

You should clean up your try-finally code.

// Original code
BufferedOutputStream writer = null;
try {
    writer = new ...;
    // use writer
} finally {
    if (writer != null) {
        writer.close();
    }
}

// Cleaner code
BufferedOutputStream writer = new ...;
try {
    // use writer
} finally {
    writer.close();
}

// Even cleaner, using try-with-resources (since Java 7)
try (BufferedOutputStream writer = new ...) {
    // use writer
}