Reputation: 34560
I'm loading a 2D array from file, it's 15,000,000 * 3 ints big (it will be 40,000,000 * 3 eventually). Right now, I use dataInputStream.readInt()
to sequentially read the ints. It takes ~15 seconds. Can I make it significantly (at least 3x) faster or is this about as fast as I can get?
Upvotes: 4
Views: 698
Reputation: 19857
Yes, you can. From benchmark of 13 different ways of reading files:
If you have to pick the fastest approach, it would be one of these:
FileChannel
with a MappedByteBuffer
and array reads.FileChannel
with a direct ByteBuffer
and array reads.FileChannel
with a wrapped array ByteBuffer
and direct array access.For the best Java read performance, there are 4 things to remember:
BufferedInputStream
).FileChannel
and MappedByteBuffer
.FileChannel
with memory mapping, or a direct
or wrapped array ByteBuffer
.Upvotes: 7
Reputation: 121860
Map your file into memory!
Java 7 code:
FileChannel channel = FileChannel.open(Paths.get("/path/to/file"),
StandardOpenOption.READ);
ByteBuffer buf = channel.map(0, channel.size(),
FileChannel.MapMode.READ_ONLY);
// use buf
See here for more details.
If you use Java 6, you'll have to:
RandomAccessFile file = new RandomAccessFile("/path/to/file", "r");
FileChannel channel = file.getChannel();
// same thing to obtain buf
You can even use .asIntBuffer()
on the buffer if you want. And you can read only what you actually need to read, when you need to read it. And it does not impact your heap.
Upvotes: 7