CODEWITHSUNDEEP
javamemory-managementgarbage-collectionheap-memory
Nick Heiner
Nick Heiner

Reputation: 122450

Java: Garbage Collection

I'm having an out of memory error. I have a large range of inputs (2^40), that is too large to hold at once. Each input is a String[].

Instead, I thought I'd run my test program on each input, write the results to a file, then discard the input. The length of the longest input is 42, so that isn't an error causing the overflow. I don't think I understand garbage collection.

PowerSet is like a lazy list - the result isn't calculated until .next() is called. Each result returns one subset of baseSet. baseSet is a String[] with length 40. runTests does some analysis on the input and writes it to a file.

PowerSet argSetSet = powerset(baseSet); 
while (argSetSet.hasNext()) {
    runTests(argSetSet.next()); //saves output to file
}

This causes an out of memory error. But I'm not saving the result of argSetSet.next() anywhere, so why should this occur? I don't need to store any data from next() or runTests(). How can I make sure it's all being garbage collected?

If I comment out runTests(), it runs fine.

Upvotes: 0

Views: 523

Answers (5)

Malaxeur
Malaxeur

Reputation: 6043

[Clippy-like icon] It appears you are calculating the powerset of a very large set. Do you want to increase your heap size?

What concerns me is that you're saying that this is a lazy list, which means that the entire powerset isn't actually in memory, but only a piece of it is in memory when you call .next(). HOWEVER, depending on what .next() actually returns (the size of the array) then the default heap size is most likely not enough.

You can increase the size of the heap by specifying -Xmx1024m (setting the maximum value size of the heap to 1gb). Obviously you can tweak that size, but this will allow you to test to see if it scales. This is not an end-all solution, but it should at least give you some runway.

Upvotes: 0

Thorbjørn Ravn Andersen
Thorbjørn Ravn Andersen

Reputation: 75376

Attach a profiler like jvisualvm and investigate where your memory goes. You might be surprised :)

Upvotes: 2

matt b
matt b

Reputation: 139931

The fact that you aren't storing the results of .next() anywhere isn't relevant, what is relevant is what .next() is actually doing.

Are you setting the heap size to a non-default size? What settings are you using to start the JVM? The default heap size of the JVM is only 64M, so one trillion entries certainly cannot fit into that space.

Upvotes: 0

Asaph
Asaph

Reputation: 162801

What's in baseSet? I'm guessing that's using up a ton of memory. This is probably exacerbated when PowerSet uses baseSet internally.

Upvotes: 0

Yishai
Yishai

Reputation: 91881

There isn't enough code to understand what is going on, primarily PowerSet, but PowerSet has to calculate the String array to return the next method. It could be that it is holding on to that object.

The memory issue is either in the runTests method or the PowerSet class. It isn't in the code you posted.

Upvotes: 4

Related Questions