CODEWITHSUNDEEP
solr
Dorin
Dorin

Reputation: 2552

reduce SOLR query time for queries executing right after indexing

I have a SOLR with 9 cores. One of the cores has around 1 000 000 documents. (~1gb size)

After finishing commit of 100 docs to this core, my next 20 -40 queries to all other cores in SOLR get very slow( 3 to 8 seconds )

I add ~100 documents every 10 minutes using Data Import Handler and then commit ( without optimize ) the index.

  1. Is there a way to reduce query time for the queries executing right after indexing ?
  2. Can it be a hardware issue taking in consideration that all other cores execute the queries slow, or is a problem the fact that I have to many cores.

I have on average 20 queries per second to all cores. I use Jetty with SOLR I have mergeFactor = 10

<mergeFactor>10</mergeFactor>

I have set autowarm count for cache

<filterCache
    class="solr.FastLRUCache"
    size="1048576"
    initialSize="131072"
    autowarmCount="943718"/>

<documentCache
  class="solr.FastLRUCache" cleanupThread="true"
  size="131072"
  initialSize="117965"
  autowarmCount="117965" />

I have 50 queries to autowarm a new searcher, for both newSearcher and firstSearcher.

<listener event="newSearcher" class="solr.QuerySenderListener">
  <arr name="queries">
    <str name="q">cake+boss</str><str name="facet">true</str><str name="facet.field">Category</str></lst>
     ...// 50 warm-up queries
  </arr>
</listener>
//the same for firstSearcher

Upvotes: 2

Views: 4005

Answers (1)

rfeak
rfeak

Reputation: 8214

You are pre-warming. This is the only way to decrease query time right after commit. You may need to work on the warming to get the right documents/queries/filters in the cache, but you are already going the right direction.

My guess is that this the rollover of the index due to commit, coupled with the cache pre-warming is sucking up all of your CPU and IO. The result is that your other cores aren't getting enough resources to respond rapidly. During a commit, your memory usage will rapidly spike, as there is a period of time where there are 2 copies of the index in Solr. Depending on your Memory usage, you could even be swapping, which would be incredibly destructive to query time.

Do a bit of CPU and Memory profiling during a commit. Make sure you aren't swapping. See if CPU cores get pegged. You most probably need to throw more hardware at this problem, or consider spreading your cores out to different machines so that they aren't impacted.

Upvotes: 3

Related Questions