CODEWITHSUNDEEP
elasticsearch
ehsan shirzadi
ehsan shirzadi

Reputation: 4859

Should I control the Index size in Elastic Search?

I have a fast growing database and I'm using Elastic Search to manage it.it has only one index and gets 200 K new documents per day. each document contains of about 5 KB text.
Should I keep using only one index or it's better to have one index for each day or something else? If so, what's the benefits of having multiple indices?

Upvotes: 0

Views: 550

Answers (1)

Alcanzar
Alcanzar

Reputation: 17155

You should definitely worry about the maximum size of your shards/index. We use daily indexes for stuff where we are inserting millions of records per day and monthly indexes where were are inserting millions per month.

A good rule of thumb is that shards should max out around 4 GB (remember there are a configurable number of shards per index).

The advantage is that when you have daily/weekly/monthly indexes, you can eventually close/delete them when your cluster becomes too big or the data isn't useful anymore. If your data is time series data, you can craft your queries to only hit the indexes that are used for the given data. Also if you've made a mistake in how many shards you really need, you can correct it going forward (because you create a new index periodically).

The disadvantage is then that you have to manage all of the extra indexes, but there are tools to do that (elasticsearch-curator for example).

Upvotes: 2

Related Questions