CODEWITHSUNDEEP
javafileio
BackSlash
BackSlash

Reputation: 22233

Storing file contents to StringBuilder apparently takes too much memory

I have this code:

public static void main(String[] args) {
    System.out.println("Reading file...");
    String content = readFile(args[0]);
    System.out.println("Done reading file.");
}

private static String readFile(String file) throws IOException {
    BufferedReader reader = new BufferedReader( new FileReader (file));
    String         line = null;
    StringBuilder  stringBuilder = new StringBuilder();
    while( ( line = reader.readLine() ) != null ) {
        stringBuilder.append( line );
    }

    return stringBuilder.toString();
}

The readFile method works fine, well, for small files.

The thing I noticed is that it takes too much memory.

If I open the System Monitor on windows (CTRL-SHIFT-ESC), I see the java process taking up to 1,8GB RAM, while the size of my file is just 550MB.

Yes, I know, loading a file entirely into memory isn't a good idea, I'm doing this just for curiosity.

The program gets stuck at Reading file... when the newly created java process starts, it takes a bunch of MB of RAM and goes up to 1,8GB.

I also tried using String concatenation instead of using StringBuilder, but I have the exact same result.

Why does it take so much memory? Is the final stringBuilder.toString causing this?

Upvotes: 0

Views: 909

Answers (1)

Peter Lawrey
Peter Lawrey

Reputation: 533530

You have to remember how these libraries work.

One byte on disk can turn into 2 byte char. The StringBuilder grows by doubling in capacity so it can be up to twice as large as you really need, and you need both the StringBuilder and String in memory at the same time.

So take your example. 550 MB can turn into 1100 MB as char alone. However, the size doubles in size so the it will be approximately the next power of two i.e. it could be 2 GB, and this is on top of a the String which would be 550 MB.

Note: the reason it is not using this much memory is that you have a bug. You are discarding all the new lines \r\n which means you have less characters.

When processing a large file where you don't have enough memory to load it into memory at once, you are better off processing the data as your read it.

BTW If you have plenty of memory, you can read the file faster, with less memory this way.

static String readFile(String file) throws IOException {
    try(FileInputStream fis = new FileInputStream(file)) {
         byte[] bytes = new byte[(int) fis.available()];
         fis.read(bytes);
         return new String(bytes);
    }
}

Upvotes: 3

Related Questions