CODEWITHSUNDEEP
javaperformanceio
Jiew Meng
Jiew Meng

Reputation: 88297

Reading large files for a simulation (Java crashes with out of heap space)

For a school assignment, I need to create a Simulation for memory accesses. First I need to read 1 or more trace files. Each contains memory addresses for each access. Example: 

0 F001CBAD
2 EEECA89F
0 EBC17910
...

Where the first integer indicates a read/write etc. then the hex memory address follows. With this data, I am supposed to run a simulation. So the idea I had was parse these data into an ArrayList<Trace> (for now I am using Java) with trace being a simple class containing the memory address and the access type (just a String and an integer). After which I plan to loop through these array lists to process them.

The problem is even at parsing, it running out of heap space. Each trace file is ~200MB. I have up to 8. Meaning minimum of ~1.6 GB of data I am trying to "cache"? What baffles me is I am only parsing 1 file and java is using 2GB according to my task manager ...

What is a better way of doing this?

A code snippet can be found at Code Review

Upvotes: 0

Views: 265

Answers (3)

Sergey Morozov
Sergey Morozov

Reputation: 4628

Try use class java.nio.ByteBuffer instead of java.util.ArrayList<Trace>. It should also reduce the memory usage.

class TraceList {

    private ByteBuffer buffer;

    public TraceList(){
        //allocate byte buffer
    }

    public void put(byte operationType, int addres) {
        //put data to byte buffer
    }

    public Trace get(int index) {
        //get data from byte buffer by index
        byte type = ...//read type
        int addres = ...//read addres
        return new Trace(type, addres)
    }

}

Upvotes: 0

MadConan
MadConan

Reputation: 3767

Like rolfl I answered your question in the code review. The biggest issue, to me, is the reading everything into memory first and then processing. You need to read a fixed amount, process that, and repeat until finished.

Upvotes: 1

rolfl
rolfl

Reputation: 17707

The answer I gave on codereview is the same one you should use here .....

But, because duplication appears to be OK, I'll duplicate the answer here.

The issue is almost certainly in the structure of your Trace class, and it's memory efficiency. You should ensure that the instrType and hexAddress are stored as memory efficient structures. The instrType appears to be an int, which is good, but just make sure that it is declared as an int in the Trace class.

The more likely problem is the size of the hexAddress String. You may not realise it but Strings are notorious for 'leaking' memory. In this case, you have a line and you think you are just getting the hexString from it... but in reality, the hexString contains the entire line.... yeah, really. For example, look at the following code:

public class SToken {

    public static void main(String[] args) {
        StringTokenizer tokenizer = new StringTokenizer("99 bottles of beer");
        int instrType = Integer.parseInt(tokenizer.nextToken());
        String hexAddr = tokenizer.nextToken();
        System.out.println(instrType + hexAddr);
    }
}

Now, set a break-point in (I use eclipse) your IDE, and then run it, and you will see that hexAddr contains a char[] array for the entire line, and it has an offset of 3 and a count of 7.

Because of the way that String substring and other constructs work, they can consume huge amounts of memory for short strings... (in theory that memory is shared with other strings though). As a consequence, you are essentially storing the entire file in memory!!!!

At a minimum, you should change your code to:

hexAddr = new String(tokenizer.nextToken().toCharArray());

But even better would be:

long hexAddr = parseHexAddress(tokenizer.nextToken());

Upvotes: 1

Related Questions