CODEWITHSUNDEEP
java
Himanshu Jain
Himanshu Jain

Reputation: 49

Best way to load a large file into arraylist in java

I have a file whose size is about 300mb. I want to read the contents line by line and then add it into ArrayList. So I have made an object of array list a1 , then reading the file using BufferedReader , after that when I add the lines from file into ArrayList it gives an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

Please tell me what should be the solution for this.

  public static void main(String[] args) {
    // TODO Auto-generated method stub
    try {
      FileReader file = new FileReader(
          "/home/dmdd/Desktop/AsiaData/RawData/AllupperairVcomponent.txt");
      ArrayList a1 = new ArrayList();
      BufferedReader br = new BufferedReader(file);
      String line = "";
      while ((line = br.readLine()) != null) {
        a1.add(line);
      }
    } catch (Exception e) {
      // TODO: handle exception
      e.printStackTrace();
    }
  }

Upvotes: 3

Views: 3461

Answers (7)

Murali
Murali

Reputation: 772

Just increase the heap size of Java

java -Xmx250m

If you running your project from IDE set -Xmx250m in arguments.

250m is 250mb

Upvotes: 4

Subhrajyoti Majumder
Subhrajyoti Majumder

Reputation: 41200

Pass -Xmx1024m to increase your heap sapce to 1024 mb.

java -Xms1024m -Xmx512m HelloWorld

You can increase up-to 4GB on a 32 bit system and on a 64 bit system you can go much higher.

Upvotes: 1

opi
opi

Reputation: 244

If you can: process the file in batches of 10000 lines or so.

read 10k lines process repeat until done

Upvotes: 0

shazin
shazin

Reputation: 21883

I agree with @Murali partly this will fix the problem you are facing. But it is advisable to use Caching when handling large files. What if the file size becomes 500Mb in a rare case. Make use of a Caching API like Memcached this will eliminate Memory Outages in JVM.

Upvotes: 0

Evgeniy Dorofeev
Evgeniy Dorofeev

Reputation: 136002

Use java.nio.file.Files.readAllLines, it returns List<String>. And if you're getting OOME increase heap size as java -Xmx1024m

Upvotes: 0

Jeff Foster
Jeff Foster

Reputation: 44706

Naively, increase the size of the heap via the Xmx command line argument (see this excellent answer for some guidance)

This'll only work up to a point though, instead consider structuring your data so that the memory requirements are minimized. Do you need the whole thing in memory at once? Perhaps you only need to test whether an item is in that set, consider using a hash or a bloom filter (etc).

Upvotes: 4

NPE
NPE

Reputation: 500277

If you have to have it in memory, you could try increasing the heap size by passing the -mx option to the java executable.

It may also be worth considering the question if you really need all that data in memory at the same time. It could be that you can either process it sequentially, or keep most or all of it on disk.

Upvotes: 2

Related Questions