CODEWITHSUNDEEP
androidmemorylucene
Ge3ng
Ge3ng

Reputation: 2430

Android Lucene OutOfMemoryExceptoin

I have a Lucene Index with 50571 documents in it from 1740 books. I have two processes that create this index. The first process is to create the index on the device document by document. This process is very slow. The other process is to create a book index on the server, (The exact same way I create it on the device) and download and merge it with the master index. This one is much quicker to create the master index. Creating the index works fine either way.

The problem is when I search on the download-merge index I get an OutOfMemoryException, but when I search with the index that was created on the device I don't get that error. I went through and created the index book by book (download-merge) and searched after each book was indexed; based on that and when I get to book ~450 I start getting the OutOfMemoryException.

What is causing me to run out of memory.

Upvotes: 2

Views: 270

Answers (1)

Ge3ng
Ge3ng

Reputation: 2430

Lucene is a memory hog. When writing "merging" indices together it stores the entire set of indices in memory twice. As quoted from the lucene documentation.

Note that this requires temporary free space in the Directory up to 2X the sum of all input indexes (including the starting index). If readers/searchers are open against the starting index, then temporary free space required will be higher by the size of the starting index

That is a lot of memory. To mitigate this we have to shrink the size of the index by calling forceMerge(int) on the index writer. It is a slow process but it does shrink the size of the index. I am call this with an argument of 1 every time there is 50 or more files in the index directory.

Upvotes: 2

Related Questions