CODEWITHSUNDEEP
javaperformancememory-mapped-files
Anna Gotkevish
Anna Gotkevish

Reputation: 57

Load text file to memory in Java

I have wiki.txt file and its size is 50 MB.

  1. I need to do several things on the file and so I thought that the best way in terms of performance is to load the file to memory, is that correct?

  2. This is the code that I written:

    File file = new File("wiki.txt");
FileInputStream fileInputStream = new FileInputStream(file);
FileChannel fileChannel = fileInputStream.getChannel();
MappedByteBuffer mapByteBuffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, file.length());
System.out.println((char)mapByteBuffer.get());

I get error on this code: mapByteBuffer.get(). I tried the get() function a few options but all of them I get error and didn't even get an error on e.getMessage() I just got null.

Another important thing to note, my text file contains English words and actions I need to do is search, if expressed is exist in this text file.

Thank you.

Upvotes: 2

Views: 5149

Answers (4)

Peter Lawrey
Peter Lawrey

Reputation: 533492

It is important to read the whole error, not just the message. Often the real information is in the exception's name not the text associated with it.

You will get an error if the file is empty as there is no first byte.

Note: the approach you are using assumes ASCII 7-bit characters. If you want to assume ISO-8859-1 characters you can use (char) (byteBuffer.get() & 0xFF)

However, if you have plan text you may find that using strings is simpler to use and not much slower. e.g. you can read a 50 MB file as text in less than a second. I would only use a memory mapped file if this is far too long.

Upvotes: 1

Olivier Croisier
Olivier Croisier

Reputation: 6149

I would suggest using a MemoryMappedFile, to read the file directly from the disk instead of loading it in memory. 

RandomAccessFile file = new RandomAccessFile("wiki.txt", "r");
FileChannel channel = file.getChannel();
MappedByteBuffer buf = channel.map(FileChannel.MapMode.READ_WRITE, 0, 1024*50);

And then you can read the buffer as usual.

Upvotes: 2

Daniel Baktiar
Daniel Baktiar

Reputation: 1712

My answers for point (1):

It depends on what you want to do with the file. If your processing doesn't involve rewind operation (looking what was read behind/before), it's best to just read as a stream and process it in one go (instead of loading all into memory).

Even if you need random access across the file, you may also be interested in doing block file operation, because your solution may not scale well when the file size change to bigger size. RandomAccessFile if you are on Java 1.4 or above. For random access, the operating system usually handles the file buffer caching quite well you don't have to handle yourself.

Upvotes: 1

Romczyk
Romczyk

Reputation: 361

I would suggest to use BufferedReader. It is much faster and requires relatively less resources. First read number of lines: 

InputStream is = new BufferedInputStream(new FileInputStream(filename));
byte[] chars = new byte[1024];
int numberOfChars = 0;
while ((numberOfChars = is.read(chars)) != -1) 
{
    for (int i = 0; i < numberOfChars; ++i) 
    {
        if (chars[i] == '\n' && numberOfChars - i != 1)
        {
            ++count;
        }           
    }
}
count++
return count; // number of lines

Then read the lines:

BufferedReader in = new BufferedReader(new FileReader(fileName));
for (int i = 0; i < endLine; i++) 
{
    String oneLine = in.readLine();
}

In this strings you can even do search for what you need.

Upvotes: 0

Related Questions