CODEWITHSUNDEEP
elasticsearchopensearch
Siva Vuyyuru
Siva Vuyyuru

Reputation: 1

open search results limitation

I am new to open search, I am working on below use case. I am using server side pagination. we are using open search as search engine, when filter query matches more than 10000 results, we are not able to fetch results. Example if we have 11000 documents matching query but when we pass page no 101 and page size 100. I am expecting to return documents from 10000 to 10100. but i am getting error on open search because of default limit of 10000. I can not increase this considering limit increase will lead to more load on cluster. Can some one assist us on how to solve this ?

Upvotes: 0

Views: 47

Answers (2)

Mike Baranczak
Mike Baranczak

Reputation: 8374

What are the requirements for your user interface? If you're OK with just showing prev/next buttons, and do not need the ability to access an arbitrary page, then search_after should do the job. (As long as you don't do anything funky with the sorting. If you're just sorting the results by one of the fields, you should be fine.)

The scroll API is not a good fit here. Scrolling is for when you need to look at every document that matches a query. It's not meant to be used for pagination.

There's a good article here discussing these topics.

Upvotes: 0

synhershko
synhershko

Reputation: 4492

Yes, this is one of Elasticsearch's query limits which obviously apply to OpenSearch as well.

By default, Elasticsearch and OpenSearch limit the number of documents returned by a search query. The size parameter controls the maximum number of results, with a default to 10 documents per query. Setting size too high can increase memory usage and require more CPU and disk work to retrieve and score the documents.

Elasticsearch has a limit of 10,000 results for a query, and trying to get beyond that will result in a “Result window is too large“ error.

For deep paging, consider using search_after or scrolling.

Upvotes: 0

Related Questions