CODEWITHSUNDEEP
javasolrlucene
Ziqi
Ziqi

Reputation: 2544

error when opening a lucene index: java.io.IOException: Map failed

I have a problem which I think is the same as that described here:

Error when opening a lucene index: Map failed

However the solution does not apply in this case so I am providing more details and asking again.

The index is created using Solr 5.3

The line of code causing the exception is:

IndexReader indexReader = DirectoryReader.open(FSDirectory.open(Paths.get("the_path")));

The exception stacktrace is:

Exception in thread "main" java.io.IOException: Map failed: MMapIndexInput(path="/mnt/fastdata/ac1zz/JATE/solr-5.3.0/server/solr/jate/data_aclrd/index/_5t.tvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 434505698 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:265)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:239)
at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:144)
at org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at uk.ac.shef.dcs.jate.app.AppATTF.extract(AppATTF.java:39)
at uk.ac.shef.dcs.jate.app.AppATTF.main(AppATTF.java:33)

The suggested solutions as in the exception message do not work in this case because I am running the application on a server and I do not have permissions to change those.

Namely,

ulimit -v unlimited

prints: "-bash: ulimit: virtual memory: cannot modify limit: Operation not permitted"

and sysctl -w vm.max_map_count=10000000

gives:"error: permission denied on key 'vm.max_map_count'"

Is there anyway I can solve this?

Thanks

Upvotes: 1

Views: 2406

Answers (1)

Ziqi
Ziqi

Reputation: 2544

I have found a solution and so I am answering myself.

If you really cannot set ulimit or vm.max_map_count, the only solution, according to http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html, is to configure your solr (or if you work with Lucene api, choose explicitly) to use SimpleFSDirectory (if windows) or NIOFSDirectory, both are slower than the default.

For example

DirectoryReader.open(new NIOFSDirectory(Paths.get("path_to_index"), FSLockFactory.getDefault()))

Upvotes: 1

Related Questions