CODEWITHSUNDEEP
elasticsearchpaginationlucene
Pawan Sharma
Pawan Sharma

Reputation: 49

How to implement pagination in ElasticSearch with multiple indexes?

We are having a lot of transactions from millions of users in our system. These transactions are stored as documents in ElasticSearch with indexes created on per month basis. For example "transactions-09-2020" is an index for September and "transactions-08-2020" is an index for August in the year 2020.

Now we need to show these transactions on a user's app, sorted in descending order on date. Since on 1st page I need to show at least 50 transactions, I might need to fetch data from multiple indexes like from current month index as well as from previous months index. User can have any number of txns in a given month. User can always scroll down from 1st page to 2nd, from 2nd to 3rd, and so on and so forth.

The main issue is to find on how many indexes do I need to search to get those 50 txns. If we go by the approach of searching in current month index and if txns are less than 50 then continue to previous months till the txns are 50, problem would be the number of hits to the ES.

We want to minimise the number of hits to fetch those 50 txns in a page. These 50 txns can belong to any page like 1st, 2nd, so on.

How to identify beforehand on how many indexes do I need to search for fetching those 50 docs? Can I have any logic to transfer this load of searching the indices to elasticsearch and me focusing only on page no. and number of txns (i.e. fetch 50 txns for page no.1, fetch 50 txns for page no. 2, fetch remaining txns for page no. 3 in this case).

Upvotes: 1

Views: 1326

Answers (1)

alexgids
alexgids

Reputation: 406

You can always create a alias like this

`POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "transactions-09-2020", "alias" : "transactions" } }
  ]
}`

and query on aliases with from and size parameter with a 'range' filter on a date field(which i assume you have it in the transaction document) In this case you don't really have to worry about which index to read from. ES will take care.

Upvotes: 1

Related Questions