CODEWITHSUNDEEP
elasticsearchpagination
Ash_s94
Ash_s94

Reputation: 817

Elasticsearch- querying data that exceeds 10k

I've recently written an application that makes queries to Elasticsearch. I've always used the "from" and "size" fields to take care of pagination. I realized today that this doesn't work for datasets greater than 10,000.

I've looked into both scroll and search_after, and although both make sense I'm not sure which I could use for my application. Here are my requirements

  1. I want to show 1000 results per page
  2. I want to be able to click NEXT and see the next 1000 results
  3. I want to click BACK and see the previous 1000 results, which I don't think scrolling allows me to do.

Upvotes: 0

Views: 222

Answers (2)

Pranav
Pranav

Reputation: 21

Another solution can be to increase max_result_window property of the index. This will allow you to fetch more data using from and size.

Beware that it will increase the heap memory usage and time taken to fetch the results as the size of the result depends on the from+size. It first fetches the from+size results and stores it in memory then returns from amount of data. So, more from+size, more heap memory usage and more time take to fetch the results.

In the documentation it says:

index.max_result_window The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory.

Upvotes: 1

ben5556
ben5556

Reputation: 3018

Yep, search_after looks like a possible solution for your use case although you need to do what is suggested here to achieve BACK pagination

Upvotes: 1

Related Questions