CODEWITHSUNDEEP
solrout-of-memorycfindex
gordon
gordon

Reputation: 1182

Solr indexing - does true for useColdSearcher help error "maxWarmingSearchers exceeded"?

cfindex of a large document set (35K docs); is there a benefit to setting useColdSearcher to true to avoid "maxWarmingSearchers exceeded" error? Running the index rebuild from CF Admin would end with no error explanation. Doing a purge and update of the entire directory was erroring: maxWarmingSearchers exceeded. I wrote a routine to get all the files and individually add them, with a dynamically increasing delay to let Solr finish each document as the index got bigger

<cfset delay=1000>
<cfdirectory action="list" directory="#dir#files" name="qFiles" >
<cfoutput query="qFiles">
    <cfindex action="update"
    collection="myColl"
    type="file"
    key="#dir#files\#qFiles.name#">
    <cfset sleep(1000+qFiles.currentRow)>
</cfoutput>

This mostly worked but would still at some point again get the maxWarmingSearchers error. I ended up having to also log the files indexed and restart the process from the last file added (along with computation to get the sleep long enough). Does temporarily setting useColdSearcher to true in the solrconfig.xml help, and is there some back door way to set that attribute in the cfindex tag or do I have to set it manually and then set it back?

Upvotes: 0

Views: 1756

Answers (1)

Nathan Johnson
Nathan Johnson

Reputation: 21

You probably want to pay more attention to your auto commit settings, as well as adjusting the commit settings of the updates themselves. Unless you're specifying settings in the solr config to "warm" the cache, using a cold cache will buy you nothing.

From the comments:

<!-- Use Cold Searcher

     If a search request comes in and there is no current
     registered searcher, then immediately register the still
     warming searcher and use it.  If "false" then all requests
     will block until the first searcher is done warming.
  -->
<useColdSearcher>false</useColdSearcher>

It doesn't sound like this will help you. You can increase maxWarmingSearchers, but most likely you need to change how often you're doing commits.

Also, keep in mind that only soft commits always a new searcher, hard commits don't necessarily. From the comments for auto commit:

<!-- AutoCommit

     Perform a hard commit automatically under certain conditions.
     Instead of enabling autoCommit, consider using "commitWithin"
     when adding documents. 

     http://wiki.apache.org/solr/UpdateXmlMessages

     maxDocs - Maximum number of documents to add since the last
               commit before automatically triggering a new commit.

     maxTime - Maximum amount of time in ms that is allowed to pass
               since a document was added before automatically
               triggering a new commit. 
     openSearcher - if false, the commit causes recent index changes
       to be flushed to stable storage, but does not cause a new
       searcher to be opened to make those changes visible.

     If the updateLog is enabled, then it's highly recommended to
     have some sort of hard autoCommit to limit the log size.
  -->

In your case, I'd recommend setting openSearcher to false if you're using autoCommit and tune the spawning of new searchers by playing with commitWithin when making the update request.

Upvotes: 1

Related Questions