CODEWITHSUNDEEP
elasticsearch
Errol Fitzgerald
Errol Fitzgerald

Reputation: 3008

Elasticsearch minimum master nodes

I have a 3 node cluster with minimum_master_nodes set to 2. If I shut down all nodes except the master, leaving one node online, the cluster is no longer operational.

Is this by design? It seems like the node that was the master shouldd remain operational, instead I get errors like this:

{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503}

All the other settings are stock and I am using the aws cloud plugin.

Upvotes: 3

Views: 3846

Answers (1)

Olly Cruickshank
Olly Cruickshank

Reputation: 6180

Yes, this is intentional.

Split brain

Imagine a situation where the other 2 nodes were still running but couldn't communicate to the the third node - you'd end up with two clusters otherwise known as a "split brain".

As the two clusters could be updating and deleting data independently of each other then recovery would be very difficult - you wouldn't have a single source of truth for the data.

By setting minimum_master_nodes to (n/2)+1 (were n is the number of nodes) you can prevent a split brain.

Single Node

If you know that the first two nodes have definitely died and not coming back - you can set the minimum_master_nodesto 1 on the remaining node (and also set to one on the other nodes before you restart them).

There is also an option no master block that lets you control what happens when you don't have a valid cluster - e.g. you could make the remaining node read-only until the cluster is re-established.

Upvotes: 6

Related Questions