CODEWITHSUNDEEP
solrlucenesolr4
ilansch
ilansch

Reputation: 4878

SOLR Response size limitation

When i query SOLR with "*" I want to know what is the maximum items i can receive from the solr.

I am sending 2 queries, in one I ask 124K rows and in one 125K, the 125K fails, I would like to understand why. I could not find anything in the SOLR logs

http://localhost:8983/solr/select?%22start=0&rows=125000&q=*&fl=UniqueId,%20entity_id,%20data_unit_id,%20score&wt=tcp&host=10.175.2.127&port=8985&queryId=4&qt=standard&debugQuery=false%22

I Receive the following error: 

HTTP ERROR 500

Problem accessing /solr/select. Reason:

{trace=java.lang.NullPointerException
,code=500}

When i am running the query: 

http://localhost:8983/solr/select?%22start=0&rows=124000&q=*&fl=UniqueId,%20entity_id,%20data_unit_id,%20score&wt=tcp&host=10.175.2.127&port=8985&queryId=4&qt=standard&debugQuery=false%22

I get good response: 

<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">6383</int>
<lst name="params">
<str name="port">8985</str>
<str name="debugQuery">false"</str>
<str name="host">10.175.2.127</str>
<str name="fl">UniqueId, entity_id, data_unit_id, score</str>
<str name="q">*</str>
<str name="queryId">4</str>
<str name="qt">standard</str>
<str name="wt">tcp</str>
<str name=""start">0</str>
<str name="rows">124000</str>
</lst>
</lst>
<result name="response" numFound="0" start="0" maxScore="0.0"/>
</response>

SOLR Logs of error when sending 125K rows query: 

INFO  - 2015-02-10 10:26:40.122; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/select params={port=8985&debugQuery=false"&host=10.175.2.127&fl=UniqueId,+entity_id,+data_unit_id,+score&q=*&queryId=4&qt=standard&wt=tcp&"start=0&rows=125000} hits=1320877 status=500 QTime=1420 
ERROR - 2015-02-10 10:26:40.123; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)

Design is to send Http request, and the response is read with TCPClient, StreamReader and NetworkSteam (C#), I dont get to reading response since exception is thrown when sending the HttpWebRequest.

Upvotes: 0

Views: 1188

Answers (1)

Mysterion
Mysterion

Reputation: 9320

Usually it's a very bad practice to set rows to that big number. For most of the applications - it should be done in a batch, like have rows = 50 and change start, when you iterate over results.

It's a very bad practice, because complexity for getting top K docs of N documents require O(K * log(N)) and in your case N = number of all docs, K - is a rows param.

Upvotes: 1

Related Questions