Performance in reading and writing files, what the best?. Serialization X Java.nio

Question

I have an object with 1 int and 4 doubles.

I compared the performance to write 5 million of these objects in a file using serialization and FileChannel object.

In the serialization used the following method to read and write the file.

    public void print() throws IOException, ClassNotFoundException{     
    ObjectInputStream input = new ObjectInputStream(new FileInputStream(this.filePath) );                   
    try {           
        while(true) {               
            this.sb = (Sbit) input.readObject();
            //System.out.println(this.sb.toString());
        } 
    }
    catch ( EOFException eofException ) {
        return; 
    } 
    catch (IOException ioException) {
        System.exit( 1 );
    }
    finally {
        if( input != null )
            input.close();
    } 

}   

public void build() throws IOException {        
    ObjectOutputStream output = new ObjectOutputStream( new FileOutputStream(this.filePath) );
    try {           
        Random random = new Random();
        for (int i = 0; i<5000000; i++) {
            this.sb = new Sbit();
            this.sb.setKey(i);
            this.sb.setXMin( random.nextDouble() );
            this.sb.setXMax( random.nextDouble() );
            this.sb.setYMin( random.nextDouble() );
            this.sb.setYMax( random.nextDouble() );

            output.writeObject(this.sb);
        }           
    } 
    catch (IOException ioException) {
        System.exit( 1 );
    } 
    finally {
        try {
            if( output != null)
                output.close();
        }
        catch ( Exception exception ) {
            exception.printStackTrace();
            System.exit(1);
        }
    }       
} 

While using java.nio was:

    public void print() throws IOException {    
    FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();  
    ByteBuffer[] buffers = new ByteBuffer[5];
    buffers[0] = ByteBuffer.allocate(4);   // 4 bytes to int
    buffers[1] = ByteBuffer.allocate(8);   // 8 bytes to double
    buffers[2] = ByteBuffer.allocate(8);    
    buffers[3] = ByteBuffer.allocate(8);   
    buffers[4] = ByteBuffer.allocate(8);   

    while (true) {
        if(file.read(buffers[0]) == -1 )       // Read the int, 
            break;                                  // if its EOF exit the loop

        buffers[0].flip();

        this.sb = new Sbit();
        this.sb.setKey(buffers[0].getInt());

        if(file.read(buffers[1]) == -1) {   // Read the int primary value
            assert false;                   // Should not get here!
            break;                          // Exit loop on EOF
        }
        buffers[1].flip();

        this.sb.setXMin( buffers[1].getDouble() );

        if(file.read(buffers[2]) == -1) {   
            assert false;                   
            break;                          
        }
        buffers[2].flip();

        this.sb.setXMax( buffers[2].getDouble() );

        if(file.read(buffers[3]) == -1) {   
            assert false;                   
            break;                          
        }
        buffers[3].flip();

        this.sb.setYMin( buffers[3].getDouble() );
        if(file.read(buffers[4]) == -1) {   
            assert false;                   
            break;                          
        }
        buffers[4].flip();

        this.sb.setYMax( buffers[4].getDouble() );

        for(int i = 0; i < 5; i++)
            buffers[i].clear();

    } 

} 


public void build() throws IOException {    
    FileChannel file = new RandomAccessFile(this.filePath, "rw").getChannel();      

    Random random = new Random();
    for (int i = 0; i<5000000; i++) {
        this.sb = new Sbit();
        this.sb.setKey(i);
        this.sb.setXMin( random.nextDouble() );
        this.sb.setXMax( random.nextDouble() );
        this.sb.setYMin( random.nextDouble() );
        this.sb.setYMax( random.nextDouble() );

        ByteBuffer[] buffers = new ByteBuffer[5];
        buffers[0] = ByteBuffer.allocate(4);   // 4 bytes to into
        buffers[1] = ByteBuffer.allocate(8);   // 8 bytes to double
        buffers[2] = ByteBuffer.allocate(8);   
        buffers[3] = ByteBuffer.allocate(8);   
        buffers[4] = ByteBuffer.allocate(8);   

        buffers[0].putInt(this.sb.getKey()).flip(); 
        buffers[1].putDouble(this.sb.getXMin()).flip();
        buffers[2].putDouble(this.sb.getXMax()).flip();
        buffers[3].putDouble(this.sb.getYMin()).flip();
        buffers[4].putDouble(this.sb.getYMax()).flip();
        try {
            file.write(buffers);
        } 
        catch (IOException e)   {
            e.printStackTrace(System.err);
            System.exit(1);
        } 

        for(int x = 0; x < 5; x++)
            buffers[x].clear();

    }
}

But I read a lot about on the java.nio and tried to use it precisely because it has better performance. But that's not what happened in my case.

To write the file were the following (java.nio):

file size: 175 MB time in milliseconds: 57638

Using serialization:

file size: 200 MB time in milliseconds: 34504

For the reading of this file, were as follows (java.nio):

time in milliseconds: 78172

Using serialization:

time in milliseconds: 35288

Am I doing something wrong in java.nio? I would like to write to the same binary files as done. There is another way to write file efficiently? actually serializing an object is the best way?

Thank you.

Answer 1

Instead of using multiple ByteBuffers, declare a single byte buffer that is large enough to hold all of the data you want to put into it. Then put data into it just like you are now. When done, flip the buffer and write it out. When you are ready to read it back in, read the data from disk into the byte buffer, flip it, and then read the data out using getInt/getDouble.

Answer 2

You are creating 25,000,000 ByteBuffer objects, with each ByteBuffer being at most 8 bytes. Thats very inefficient.

Create just one ByteBuffer by allocating it to 38 bytes outside the loop (before the for statement)

Inside the loop you can use the same ByteBuffer as follows:

  buffer.clear();

  buffer.putInt(this.sb.getKey()); 
  buffer.putDouble(this.sb.getXMin());
  buffer.putDouble(this.sb.getXMax());
  buffer.putDouble(this.sb.getYMin());
  buffer.putDouble(this.sb.getYMax());

  buffer.flip();

  try
  {
     file.write(buffer);
  }
  catch (IOException ex)
  {
     ex.printStackTrace();
     //etc...
  }

  buffer.flip();

Try it out and let us know if you see any improvements.

Answer 3

I haven't tried to serialize stuff on my own, but have achieved good results with kryo. It is a lot faster than standard Java serialization.