CODEWITHSUNDEEP
elasticsearchlucene
Andrzej Wisłowski
Andrzej Wisłowski

Reputation: 123

How many shards for elasticsearch node

I have got an elasticsearch cluster with setup of 6 nodes (version 2.3.5, 16 GB heap).

I have got there over 3000 timeseries indices with default 5 shards and one replica. Lastly I have a problem with nodes connecting to the cluster. Not only 4 nodes are present. The other two cannot connect. They log an error connecting to the master node:

Caused by: org.jboss.netty.channel.ConnectTimeoutException: connection timed out:

So on 4 nodes there are over 3000 shards on each node. Each node have over 26000 open descriptors.

I started reindexing them to have less shards. But the question is: How many shards is a limit for typical elasticsearch node

Upvotes: 2

Views: 668

Answers (1)

pbo
pbo

Reputation: 248

As far as i know, you should avoid having more than 150 shards per node (and that's already a lot !).

So, I'd suggest you to not have 3000 indices, and try to put everything in a smaller number of indices. Would it be possible for you to add an attribute to segregate the different types of document you have in your 3000 idx ?

But maybe you need that much indices. How many documents do you have in each index ?

Upvotes: 1

Related Questions