CODEWITHSUNDEEP
javaout-of-memory
dotwin
dotwin

Reputation: 1332

java.lang.OutOfMemoryError even though plenty of

I am trying to read a 2.5GB txt file into my application. I am running Win7 x64 and have 43GB of mem available (out of 64GB). I tried playing around with -Xmx -XX:MaxParmSize -XX:ParmSize etc. None of these affect the error. What else could I try? This error seems very odd as I certainly have enough heap space available.

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuilder.append(Unknown Source)
    at j.utilities.IO.loadString(IO.java:187)
    at j.utilities.IO.loadString(IO.java:169)
    at city.PreProcess.main(PreProcess.java:78)

I am running

java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

Thanks a lot in advance.

============== ANSWER ==============

OK, I just tested it with

StringBuilder sb = new StringBuilder();
for ( int i=1; i<Integer.MAX_VALUE; i++ )
    sb.append("x");

and got

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
...

Thus, it really is StringBuilder which tries to build an array bigger than Integer.MAX_VALUE.

In case interested

StringBuilder sb = new StringBuilder();
int i=1;
try {
    for ( ; i<Integer.MAX_VALUE; i++ )
        sb.append("x");
} catch ( OutOfMemoryError e ) {
    System.out.println(i); // OUTPUT: 1207959551
    System.out.println(Integer.MAX_VALUE); // OUTPUT: 2147483647
}

With StringBuilder you can accumulate 1,207,959,550 chars - far less than Integer.MAX_VALUE.

Upvotes: 7

Views: 30643

Answers (4)

Raj
Raj

Reputation: 1724

You can hold the data in string buffer data is List<String> in define interval and clear the StringBuffer.

Upvotes: 0

Diego Basch
Diego Basch

Reputation: 13059

You could create a new StringBuilder with a size, e.g.

    StringBuilder sb = new StringBuilder(Integer.MAX_VALUE);

The problem is that you're trying to read a file that's larger than what StringBuilder can have in its array. You have several options, e.g.:

1) Do you really need to read the entire file into memory at once? If so, you'll have to read it into several StringBuilders.

2) Process the file sequentially.

3) Read it into a compressed structure, and uncompress the parts you need when you need them.

Upvotes: 3

reprogrammer
reprogrammer

Reputation: 14718

You're trying to allocate an array that is too large. This is because you're trying to create a very long String. Since arrays are indexed by an integer, an array cannot have more than Integer.MAX_VALUE elements. Even if the your heap size is very large, you won't be able to allocate an array that has more than Integer.MAX_VALUE elements, simply because you cannot index its elements using an Integer. See Do Java arrays have a maximum size? for more details.

Upvotes: 10

Bryan Glazer
Bryan Glazer

Reputation: 852

You should look into the -Xmsn option for the java command.

It specifies the initial size of the memory allocation pool.

Edit: I see you've already done that.

Upvotes: 0

Related Questions