CODEWITHSUNDEEP
amazon-web-serviceselasticsearchaws-elasticsearch
Kumaran
Kumaran

Reputation: 21

AWS Elasticsearch - Suggest how many number shard & replica create for m4.large.elasticsearch instance

I have 3 master node and 5 data node on this instance type m4.large.elasticsearch (2cCPU & 8gb memory) with storage of 512gb.

Please suggest max number of shard and replica i can create for above configuration.

Upvotes: 0

Views: 753

Answers (1)

Prabuddha Chakraborty
Prabuddha Chakraborty

Reputation: 528

You may have as many numbers of shards and replica depending upon your volume size and usage.

Replicas are primarily for search performance, and a user can add or remove them at any time. They give you additional capacity, higher throughput, and stronger failover. It is always recommend a production cluster to have 2 replicas for failover. Also note doubling the number of replicas will also double your disk space usage.

The number of shards you can hold on a node will be proportional to the amount of heap you have available, but there is no fixed limit enforced by Elasticsearch. A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600-750 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health. After you configure an Elasticsearch cluster, it's critically important to realize that you cannot modify the shard allocation later. If you later find it necessary to change the number of shards, then you would need to reindex all the source documents. (Although reindexing is a long process, it can be done without downtime).

Upvotes: 2

Related Questions