CODEWITHSUNDEEP
elasticsearchelasticsearch-5
user_78361084
user_78361084

Reputation: 3928

Error 429 [type=reduce_search_phase_exception]

I have many languages for my docs and am following this pattern: One index per language. In that they suggest to search across all indices with the

/blogs-*/post/_count

pattern. For my case I am getting a count across the indices of how many docs I have. I am running my code concurrently so making many requests at same time. If I search

/blogs-en/post/_count

or any other language then all is fine. However if I search 

/blogs-*/post/_count

I soon encounter: 

"Error 429 (Too Many Requests): [reduce]  [type=reduce_search_phase_exception]
"

Is there a workaround for this? The same number of requests is made regardless of if I use 

/blogs-en/post/_count or /blogs-*/post/_count.

I have always used the same number of workers in my code but re-arranging the indices to have one index per language suddenly broke my code.

EDIT: It is a brand new index without any documents when I start the program and when I get the error I have about 5,000 documents so not under any heavy load.

Edit: I am using the mapping found in the above-referenced link and running on a local machine with all the defaults of ES...in my case shards=5 and replicas=1. I am really just following the example from the link.

EDIT: The errors are seen with as few as 13-20 requests are made and I know ES can handle more than that. Searching /blogs-en/post/_count instead of /blogs-*/post/_count, etc.. can easily handle thousands with no errors.

Another Edit: I have removed all concurrency but still can only access 40-50 requests before I get the error.

Upvotes: 1

Views: 2087

Answers (1)

Mohammad Mazraeh
Mohammad Mazraeh

Reputation: 1074

I don't get an error for that request and it returns total documents.
Is you'r cluster under load?
Anyway, using simple aggregation you can get total document count in hits.total and per index document count in count_per_index part of result:

GET /blogs-*/post/_search
{
    "size": 0, 
   "query": {
      "match_all": {}
   },
   "aggs": {
      "count_per_index": {
         "terms": {
            "field": "_index"
         }
      }
   }
}

Upvotes: 1

Related Questions