CODEWITHSUNDEEP
performanceelasticsearch
Yu Hao
Yu Hao

Reputation: 122373

Big single shard vs many shards

There are two best practices on ElasticSearch shards:

In my case, they are somewhat controversial. To be more specific, assuming the index size is 2 TB, and there are 10 nodes. How many shards should I config:

Option 1: 10 shards with 200GB each

or

Option 2: 40 shards with 50GB each

Which is the better option for query latency performance?

Upvotes: 2

Views: 2417

Answers (2)

Harald
Harald

Reputation: 5093

To add to Val's answer: more shards allow for smoother shard distribution in case you want to add nodes for better performance. 10 shards on 10 nodes does not allow to distribute shards to additional nodes. 40 shards would allow to scale up easily with more nodes.

Further, should the disk space get tight, smaller shards may still allow Elasticsearch to move shards to-and-fro, because it needs at least the room for one more shard to do anything.

Upvotes: 0

Val
Val

Reputation: 217254

Whatever is deemed "optimal" is usually only optimal in theory, in practice you need to make some trade-offs. Most of the time you'll certainly want to have at least one replica per primary shard (fault tolerance), so you'd have at least 2 shards per node (unless you have 5 primary shards @ 400GB each). So much for optimality, let's get down to earth...

You didn't mention the amount of heap per node, but since you shouldn't cross the 30.5GB heap limit per node, you should clearly lean towards shards having at most ~50GB of data. 50 shards @40GB would work, too.

I wouldn't try 200GB shards as that's probably too big. I also wouldn't try having 1000 2GB shards as there would be too many shards.

Ultimately, it depends on your use case and your hardware. Is your index undergoing a heavy search load, or is it mainly handling indexing requests? How many concurrent search/index requests does your cluster need to handle? The best way to know is to test all that, but without more information, the second option is clearly better than the first one. And don't forget you probably need one replica per primary shard, too, which would double your storage needs (i.e. 400GB per node)

Upvotes: 3

Related Questions