java - limit of ArrayList<String>

Question

I am trying to read a text file and store every line into ArrayList However, the text file is too long (about 2,000,000) lines and error: java.lang.OutOfMemoryError occurs.

How do i know if the arraylist is full and then create another arraylist to store the remaining data automatically?

Sorry for my poor english.

Thanks for your help.

Answer 1

As the other answers suggested, your problem is because you run out of memory before your ArrayList is full. If you still don't have enough memory after increasing the heap space size, BigArrayList will solve your problems. It functions like a normal ArrayList and automatically handles swapping data between disk and memory. Note that the library currently supports a limited number of operations, which may or may not be sufficient for you.

Answer 2

2 million lines is far beyond the maximum size for Java Collection (INTEGER.MAX_VALUE or 2 billion indexes).

You are more likely to have heap space outOfMemory error. You can do either

1. Increase your JVM maximum heap memory allocation.

java -Xmx4g

4g = 4GB.

The default maximum heap size is half of the physical memory up to a physical memory size of 192 megabytes and otherwise one fourth of the physical memory up to a physical memory size of 1 gigabyte.

http://www.oracle.com/technetwork/java/javase/6u18-142093.html

2. as konsolas recommends, read line by line and store it into a file and flush the variable.

Hope it helps!

Answer 3

This depends on what you are planning to do with the file. You're definitely not going to be able to store all of it in memory, as shown by your error.

Depending on what you're trying to do with the file, processing it in blocks and then saving it would be a better idea. For example:

• Read the first 1000 lines of the file
• Process these lines/save into another file, etc.
• Read the next 1000 lines
• etc.

An ArrayList can theoretically hold 2,147,483,647 items. (max int)