CODEWITHSUNDEEP
apacheluceneout-of-memory
Sanchit Khera
Sanchit Khera

Reputation: 1005

OutOfMemory in Apache Lucene

I'm getting OutOfMemory error in Apache Lucene.

Here is the problem code:

DirectoryReader oldReader = directoryReader;
DirectoryReader newReader = DirectoryReader.openIfChanged(directoryReader);
if ((newReader != null) & (oldReader != newReader)) {
   directoryReader = newReader;
}

Here is the log:

Caused by: java.lang.OutOfMemoryError: Java Heap Space
    at java.lang.Class.getMethodImpl(Native Method)
    at java.lang.Class.getMethod(Class.java:917)
    at org.apache.lucene.store.MMapDirectory$MMapIndexInput$1.run(MMapDirectory.java:244)
    at org.apache.lucene.store.MMapDirectory$MMapIndexInput$1.run(MMapDirectory.java:241)
    at java.security.AccessController.doPrivileged(AccessController.java:327)
    at org.apache.lucene.store.MMapDirectory$MMapIndexInput.freeBuffer(MMapDirectory.java:241)
    at org.apache.lucene.store.ByteBufferIndexInput.close(ByteBufferIndexInput.java:295)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:788)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:694)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:400)
    at org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:349)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:303)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
    at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:254)
    at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)

Any idea, what can be the problem?

Upvotes: 2

Views: 381

Answers (1)

Mysterion
Mysterion

Reputation: 9320

Since you're using MMapDirectory, you should be aware, that:

NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult MMapDirectory(Path, LockFactory, int) if you have problems with mmap failing because of fragmented address space. If you get an OutOfMemoryException, it is recommended to reduce the chunk size, until it works.

MMapDirectory uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space

So, consider tuning your chunk size or choose other implementation of the Directory, like NIOFSDirectory or SimpleFSDirectory

Upvotes: 1

Related Questions