Elasticsearch cluster initialization

Question

I just setup a 3 node Elasticsearch cluster, with each node having common settings (pasted at the end of the post)

However, when I start my master node, and try to get the cluster status or even check if any one of the nodes is up, I get a 503 as the status code. Also, shutdowns (on any of the nodes) do not work.

Could someone please tell me what I'm doing wrong here? The log file on Node 1 says:

[ESNode1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]

Here's snippets from the elasticsearch.yml config files:

Node 1

cluster.name: myCluster

node.name: ESNode1

node.master: true

node.data: true

discovery.zen.minimum_master_nodes: 2

discover.zen.ping.timeout: 20s #just for good measure

discovery.zen.ping.multicast.enabled: false

Node 2

cluster.name: myCluster

node.name: ESNode2

node.master: true

node.data: true

discovery.zen.minimum_master_nodes: 2

discover.zen.ping.timeout: 20s

discovery.zen.ping.multicast.enabled: false

Node 3

cluster.name: myCluster

node.name: ESNode3

node.master: false

node.data: true

discovery.zen.minimum_master_nodes: 2

discover.zen.ping.timeout: 20s

discovery.zen.ping.multicast.enabled: false

Thank you!

Answer 1

Disabling multicast discovery means discovery pings will only be sent to specific addresses. The addresses/hosts are those specified in discovery.zen.ping.unicast.hosts.

Note that a single address can be specified. When a node joins it becomes aware of all nodes in the cluster, and can start communicating with them directly.

To clarify using Jettros' example: discovery.zen.unicast.hosts:["127.0.0.1:9300"] will cause the nodes bound to 9301 and 9302 to ping only 9300.

If 9301 joins first it 'already knows' all other nodes in the cluster (just 9300).

If 9302 subsequently joins it will become aware of 9301 and vice-versa. If 9301 and 9302 cant join with 9300 the cluster will not be formed.

Answer 2

You configure that the minimum master nodes is 2. This means your cluster needs at least two master nodes. This is fine, however, together with the setting discovery.zen.ping.multicast.enabled: false this is hard to get working. This setting means you are not going to look for other nodes. So you should configure the nodes manually using the setting hosts.

You can find more information here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#unicast

An example for three nodes running on one machine: discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300","127.0.0.1:9301","127.0.0.1:9302"]