CODEWITHSUNDEEP
solrindexingfull-text-searchreal-timesunspot-rails
Sid
Sid

Reputation: 6264

Sunspot with Solr 3.5. Manually updating indexes for real time search

Im working with Rails 3 and Sunspot solr 3.5. My application uses Solr to index user generated content and makes it searchable for other users. The goal is to allow users to search this data as soon as possible from the time the user uploaded it. I don't know if this qualifies as Real time search.

My application has two models

  1. Posts
  2. PostItems

I index posts by including data from post items so that a when a user searches based on certain description provided in a post_item record the corresponding post object is made available in the search.

Users frequently update post_items so every time a new post_item is added I need to reindex the corresponding post object so that the new post_item will be available during search.

So at the moment whenever I receive a new post_item object I run 


 post_item.post.solr_index! #

which according to this documentation instantly updates the index and commits. This works but is this the right way to handle indexing in this scenario? I read here that calling index while searching may break solr. Also frequent manual index calls are not the way to go.

Any suggestions on the right way to do this. Are there alternatives other than switching to ElasticSearch

Upvotes: 3

Views: 863

Answers (2)

so_mv
so_mv

Reputation: 3998

If you are just starting out and have the luxury to choose between Solr and ElasticSearch, go with ElasticSearch.

We use Solr in production and have run into many weird issues as the index and search volume grew. The conclusion was Solr was built/optimzed for indexing huge documents(word/pdf content) and in large numbers(billions?) but updating the index once a day or a couple of days when nobody is searching.

It was a wrong choice for consumer Rails application where documents are small, small in numbers( in millions) updates are random and continuous and the search needs to be somewhat real time( a delay of 5-10 sec is fine).

Some of the tricks we applied to tune the server.

removed all commits (i.e., !) from rails code, 
use Solr auto-commit every 5/20 seconds, 
have master/slave configuration, 
run index optimization(on Master) every 1 hour 
and more.

and we still see high CPU usage on slaves when the commit triggers. As a result some searches take a long time(> 60 seconds at times).

Also I doubt if the batching indexing sunspot_index_queue gem can remedy the high CPU issue.

Upvotes: 1

okliv
okliv

Reputation: 3959

try to use this gem https://github.com/bdurand/sunspot_index_queue

you will than be able to batch reindex, let's say, every minute, and it definitely will not brake an index

Upvotes: 1

Related Questions