CODEWITHSUNDEEP
javaconcurrencylucene
user193116
user193116

Reputation: 3568

How to speed up a concurrent Lucene application?

My users are submitting document to my application which updates the index and other users can query that index. Unlike crawlers, the indexing is intermittent -- only index when a user submits a document. I expect more Index Searches than Index write operations. ( I am not using Solr because that is an overkill for my need) and no deletes or updates of indexed documents.

This is my code for adding document to Index and reading from the Index. How can I optimize it further ?

public void addDocument(Document doc) throws CorruptIndexException, LockObtainFailedException, IOException{

    IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer(Version.LUCENE_35)).setIndexDeletionPolicy(NoDeletionPolicy.INSTANCE).setRAMBufferSizeMB(256.0);
    TieredMergePolicy tmp = new TieredMergePolicy();
    tmp.setUseCompoundFile(false);
    tmp.setMaxMergedSegmentMB(1000000.0);
    //tmp.setReclaimDeletesWeight(3.0);
    //tmp.setMaxMergedSegmentMB(7000.0);
    iwc.setMergePolicy(tmp);
    // Make sure merges run @ higher prio than indexing:
    final ConcurrentMergeScheduler cms = (ConcurrentMergeScheduler) iwc.getMergeScheduler();
    cms.setMergeThreadPriority(Thread.currentThread().getPriority() + 2);
    cms.setMaxThreadCount(1);
    cms.setMaxMergeCount(4);



    IndexWriter iw = new IndexWriter(directory, iwc);
    iw.addDocument(doc);
    iw.close();
}

When a search request comes in, I create a new IndexSearcher as follows

public IndexSearcher getIndexSearcher() throws CorruptIndexException, IOException {

        IndexSearcher is= new IndexSearcher(IndexReader.open(directory, false));
        return is;
    }

// I then use the searcher for actual queries not shown here

Upvotes: 1

Views: 885

Answers (1)

Robert Muir
Robert Muir

Reputation: 3195

Keep IndexWriters and IndexReader around. Don't open a new writer and close it just to add a single document. Don't open a new reader just to respond to one query, use SearcherManager or NRTManager instead.

Upvotes: 10

Related Questions