CODEWITHSUNDEEP
javaarraylistjava.util.scannerheap-memory
trentjones21
trentjones21

Reputation: 571

Out of memory using Scanner to read large file into memory

The following block of code is throwing a java.lang.OutOfMemoryError exception when I pass a large file into the scanner. What is the best way to fix this? Is the problem in the arraylist or the scanner?

ArrayList rawData = new ArrayList();
Scanner scan = new Scanner(file);

while (scan.hasNext()) {
    String next = scan.next();
        rawData.add(next);
}

Upvotes: 0

Views: 4096

Answers (4)

grepit
grepit

Reputation: 22392

The main problem is storing on the array list. Also, try to use the bufferReader and just do the processing inside the while statement instead of trying to add it to the arraylist. Here is a simple example.

        File file = new File("C:\\custom_programs\\reminder_list.txt");
        BufferedReader br = new BufferedReader(new FileReader(file));
        String line;
        while ((line = br.readLine()) != null) {
            // do something with line.
            System.out.println(line);
        }
        br.close();

Upvotes: 3

Sotirios Delimanolis
Sotirios Delimanolis

Reputation: 280030

The default separator for a Scanner is whitespace.

public Scanner(ReadableByteChannel source) { // Your File is converted to a ReadableByteChannel from another constructor 
    this(makeReadable(Objects.requireNonNull(source, "source")),
         WHITESPACE_PATTERN);
}

So if your file contains many whitespace characters, you will loop so many times at

while (scan.hasNext()) {
    String next = scan.next();
    rawData.add(next);
}

putting in many objects in your ArrayList but not garbage collecting anything (ie. not freeing memory).

Each call to next() returns the next token until a whitespace is found. Either change the separator, increase your memory size, or change your design.

What is your file format?

Upvotes: 0

RandomQuestion
RandomQuestion

Reputation: 6998

Instead of loading all the rows from file into ArrayList, do the operation you want to on each record as soon as you read it. Loading whole file into memory would cause an OOM issue if heapsize is not large enough.

Scanner scan = new Scanner(file);
while (scan.hasNext()) {
    String next = scan.next();
    //do what you want to do on next
}

Upvotes: 0

Clyde
Clyde

Reputation: 7549

Increase the java heap size, e.g.

java -Xmx6g myprogram

will set the heap size to 6 gigabytes. Of course there will always be a limit....

Upvotes: 3

Related Questions