CODEWITHSUNDEEP
luceneindexingelasticsearch
gakhov
gakhov

Reputation: 1971

Elasticsearch multindex performance

I'm thinking about moving from one index to day-based indexes (multi-index) in our Elasticsearch cluster with huge number of records.

The actual question is how it can affect the performance of indexing, searching and mapping in the ES cluster?

Is it take more time to search through one huge index than from a hundreds of big indices?

Upvotes: 1

Views: 267

Answers (2)

gakhov
gakhov

Reputation: 1971

Finally, we have implemented ES multi-indexing in our company. For our application we chose monthly indices strategy, so we create a new index every month.

Of course, as it was advised by @femtoRgon, the search through all smaller indices takes a little bit more, but speed of application has been increased because of its logic.

So, my advice to everybody who wants to move from one index to multi-indices: make research of your application needs and select appropriate slices of the whole index (if it's really needed).

As example, i can share some results of research of our application, that helped us to make a decision to use monthly indices:

  • 90-95% of our queries are only for last 3 months
  • we have about 4 big groups of queries: today, last week, last month and last 3 months (of course, we could create weekly or daily indices, but they would be too small, since we don't have enough documents inside)
  • we can explain to customers why he need to wait if he makes "non usual" query across whole period (all indices).

Upvotes: 1

femtoRgon
femtoRgon

Reputation: 33351

It will take less time to search through one large index, rather than hundreds of smaller ones.

Breaking an index in this fashion could help performance if you will be primarily searching only one of the broken out indexes. In your case, if you most often will need to search for records added on a particular day, then you might be helped by this, performance-wise. If you will mostly be searching across the entire of range of indexes, you would generally be better off searching in the single monolithic index.

Upvotes: 2

Related Questions