CODEWITHSUNDEEP
javasolr
user850234
user850234

Reputation: 3463

Solr search query time increases as the start keeps on increasing

I have currently over 25 million documents in Solr and the volume will gradually increase. I need to search for the records over such big size of Solr indexes. The query response time is low when the start is low, e.g 0. But as the start increases, e.g 100000 , searching in Solr is also taking time. How can i make the search faster even with high start number over large data sets in Solr? The rows remain constant only the start keeps on increasing. I don't want the response time to increase as the start keeps on increasing instead want the result returned for start=100000 should take the same time as for start=0 with say suppose rows=1000 as this is performance issue. Any help would be appreciated.

Upvotes: 4

Views: 1558

Answers (2)

cheffe
cheffe

Reputation: 9500

With the release of Solr 4.7 a new feature has been introduced Cursors. This has been done exactly to address the problem of Deep Paging. If you still have the problem and you may perform the upgrade to Solr 4.7 this is the best option for you.

Some references about deep paging with Solr

Upvotes: 2

cheffe
cheffe

Reputation: 9500

The problem you are facing is called Deep Paging. There is a good article about it on solr.pl and an incomplete issue on Solr's tracker.

The solution mentioned in the article will require you to sort your result, if that is not feasible for you the solution will not work. The idea is to sort by a stable attribute, in the article that is price and then filter with a price range, like fq=price:[9000+TO+10000].

If you combine that fq with a suitable start - like start=100030 - you will get better performance, as solr will not collect the documents that do not match the fq.

But you will need to make at least one query in advance to fetch the suitable meta data, like how many docs have been found at all.

Upvotes: 3

Related Questions