CODEWITHSUNDEEP
solrlucenetagsinformation-retrievaledismax
swami
swami

Reputation: 733

Find similar results with Lucene / SOLR index

We have an application for tagging user selections over a large corpus of MS Word documents. We tag these selections with one or more keyword tags, and usually a title tag. We want to add a feature where the selected text is instantly analyzed, and the tagger is presented with a list of most-likely keyword and title tags (based on the existing tagged text selections)

We are using a SOLR index. I have been told that we can simply issue the selected text as the query itself to return similar selections. However, the selected text could be anywhere between 200 and 6000 words long. A 6000 word query may be a problem in terms of memory usage!

I thought we could do some very aggressive stopword removal to significantly reduce the number of words in the queries, leaving only the very meaningful words. We have been working with this corpus for the last 10 years and we are very familiar with the subject matter and the vocabulary used, so this would be easy for us to do. But the problem is that we also use the same index for allowing the normal users to search the index, and if we remove too many common words, then their normal queries may not work properly (especially phrase queries).

We would also like to boost the results that contain the text of the query within a smaller range, rather than just spread arbitrarily throughout the document.

Another issue is that we allow nested selections. The outer selection may be more general in nature and be around 5000 words long, and the inner selections will be shorter and topically more specific. However, since both selections contain the same text, SOLR ranks them both highly, when the outer selection may not be so relevant

I have spent the last few days going through the SOLR query parser documentation, and it looks like this should be doable, but I'm still not sure exactly what I need to do to make this work. Any suggestions would be much appreciated.

Upvotes: 0

Views: 151

Answers (1)

Hariharan
Hariharan

Reputation: 881

Solr have multi-core facility. So if you can have one core for your internal work and you can reveal the other core for public domain, it may solve your issue. You can refer this section http://wiki.apache.org/solr/Solr.xml%20(supported%20through%204.x) or you can refer Solr cores and solr.xml section in solr reference manual.

Upvotes: 0

Related Questions