CODEWITHSUNDEEP
elasticsearch
billrichards
billrichards

Reputation: 2060

ElasticSearch index has thousands of shards

A request to the _search endpoint returns the following:

{
    "took": 303,
    "timed_out": false,
    "num_reduce_phases": 12,
    "_shards": {
        "total": 6121,
        "successful": 6121,
        "skipped": 0,
        "failed": 0
    },

The number of shards is increasing periodically. What could cause the number of shards to increase like this? I do not see any evidence of the shards being explicitly added.

UPDATE The number of shards was for all indexes, not this index in particular. The high number of shards was simply due to adding several daily log indexes without a proper delete policy.

Upvotes: 0

Views: 297

Answers (1)

llermaly
llermaly

Reputation: 2500

Shards are set up on index creation, depending on the elasticsearch version the defaults may change.

For the last version it is 1 primary shard and 1 replica, meaning each index will use 2 shards. You can change the number of replicas after creating the index, but not the number of primaries.

Then for example if you have 2 primary shards and 3 replicas index you will have a total of 8 shards ( 2 pri + 2 pri * 3 rep = 8)

Common reasons of seeing shards increasing it is you are ingesting 1 index a day. Many ingestion tools will use the current day as index name, causing a new index generating every day.

You can read more about shards here:

https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

Upvotes: 1

Related Questions