Distributed Search Across Multiple Solr Instances

Question

I have 100 billion rows of data that I have split into multiple solr instances, each with a separate schema.

I need to:

• Query each instance.
• Get the results from each instance.
• Append those results to a final query.
• Call a final Solr instance for the ultimate result.

How can I do this? Do I need to write a separate requestHandler?

eg,

$ curl http://localhost:8983/solr/select?q=query1.result AND ... AND queryN.result

Answer 1

You can use a combination of shards and filter queries:

• use a single solr which does a sharded search using the shards parameter: http://wiki.apache.org/solr/DistributedSearch

and then

• do a filter query on the result within the same query http://wiki.apache.org/solr/CommonQueryParameters#fq

Example:

your local 'combination solr' is running on localhost:8983 and the other solrs are running on host1:8983, host2:8983, ... You are searching remotely on these peers for 'field1:query1' but you want to filter out of the results the query 'field2:query2'. So you call:

http://localhost:8983/solr/select?shards=host1:8983/solr,host1:8983/solr&q=field1:query1&fq=field2:query2

Answer 2

I think DirectSolrConnection could help. I have similar requirement like yours, and I did use DirectSolrConnection for it.

Answer 3

i had done this by solrj (For instance solution)

 File home = new File("C:\\workspace\\SolrMultipleCore\\solr");
 File f = new File( home, "solr.xml" );
 CoreContainer container = new CoreContainer();
 container.load( "C:\\workspace\\SolrMultipleCore\\solr", f );
 EmbeddedSolrServer server = new EmbeddedSolrServer( container,"core1");
 EmbeddedSolrServer server1 = new EmbeddedSolrServer( container,"core2");
 String query=params.getParams("q");
 String query1=params.getParams("q1");
 SolrQuery solrquery=new SolrQuery(query);
 QueryResponse q = server.query(solrquery);
 QueryResponse q1 = server1.query(solrquery);

Solr.xml

<solr persistent="true">
 <property name="snapshooter" value="C:\solr1\bin\snapshooter.sh" />
 <cores adminPath="/admin/cores">
  <core name="core1" instanceDir="core0"/>   
  <core name="core2" instanceDir="core1" />
 </cores>
</solr>

Still i am making research on how to do this inside solr.

Let me know if there is any details need on this

Answer 4

what you are looking for is called distributed search -> http://wiki.apache.org/solr/DistributedSearch