Why is it recommended to create clusters with odd number of nodes

Question

There are several resources about distributed systems, like the mongo db documentation that recommend odd number of nodes in a cluster.

What are the benefits of having odd number of nodes?

Answer 1

The odd number of nodes help and is not necessary to elect a leader in a cluster. It is essential to avoid multiple leaders getting elected, a condition known as split-brain problem. consensus algorithms use voting for electing the leader. i.e, elect the node with majority votes.

consider a cluster of 5 nodes. the majority required is 3 (5/2 or 2 + 2 + 1 - the deal breaker).

It is important to note that the majority of the cluster votes are required for leader election even under failure conditions.

consider 1 out of 5 nodes failed. we can still elect a leader with majority votes of 3. well, what if out of the 4 nodes, two nodes receive equal votes of 2? that's left to the consensus algorithm to resolve the contention (maybe, simply initiate re-election). This is not the problem a odd cluster solves. when the number of active nodes are even, there is a small overhead to achieve consensus. Again, this cannot be avoided with odd/ even number of nodes since under failure conditions, the nodes can be odd or even. One usually gets confused about achieving majority when one of the odd nodes fail, leaving them even in number. it should be clear by now, that a tie is not a problem and a majority is sufficient for leader election.

when 2 out of 5 nodes fail, we can still elect a leader with majority votes of 3 i.e when all 3 available nodes vote for the same node. The cluster cannot tolerate more than 2 node failures since majority cannot be achieved.

another point to add here is, how the odd number helps in case of network partitions. In the worst case, a network partition can split the cluster into exactly two equal halves which cannot happen in an odd-numbered cluster.

why does the leader election require the majority of the initial cluster size and not the majority of active nodes? say, 2 nodes failed in a 7-node cluster, leaving 5 nodes running. what is the problem if a leader is elected with majority of 3 votes? the problem is you cannot assure that the two nodes are actually down or only partitioned. This is the very purpose of leader election, to avoid the split brain problem. A 7-node cluster could be partitioned into two parts with 3 and 4 nodes. And, two leaders could be elected on either side. a majority of 7/2 = 4 elects a single leader.

As long as the number of operating/ reachable nodes are greater or equal to floor(n/2)+1, to reach consensus based on majority w.r.t the initial cluster size, the cluster can continue to operate

Answer 2

Short Answer: Higher Fault Tolerance.

This is a general principle that applies to many other clusters that uses RAFT alike leader election algorithms such as Kubernetes ETCD clusters.

If it uses RAFT for leader selection, cluster needs a majority of nodes, a quorum, to agree on a leader. For a cluster with n members, quorum is (n/2)+1.

In terms of fault tolerance, adding an additional node to an odd-sized cluster decreases the fault tolerance. How? We still have the same number of nodes that may fail without losing quorum however we have more nodes that can fail which means possibility of losing quorum is actually higher than before.

For fault tolerance please check this official etcd doc for more information.

Answer 3

Short answer: in this case of MongoDB, having an odd number of nodes increases your clustered system's availability (uptime).

Look at the table in the MongoDB documentation you linked:

+-------------------+------------------------------------------+-----------------+
| Number of Members | Majority Required to Elect a New Primary | Fault Tolerance |
+-------------------+------------------------------------------+-----------------+
|         3         |                    2                     |        1        |
+-------------------+------------------------------------------+-----------------+
|         4         |                    3                     |        1        |
+-------------------+------------------------------------------+-----------------+
|         5         |                    3                     |        2        |
+-------------------+------------------------------------------+-----------------+
|         6         |                    4                     |        2        |
+-------------------+------------------------------------------+-----------------+

Notice that how when you have an odd number of members and add one more (to become even), your fault tolerance does not go up! (Meaning, your cluster cannot tolerate more failed members than it originally could)

This is because MongoDB requires a majority of members to be up in order to elect a primary. This property is not specific to MongoDB, but any clustered system that requires a majority of members to be up (for example, see also etcd).

Your system availability actually goes down when increasing to an even number of nodes because, although your fault tolerance remains the same, there are more nodes that can fail so the probability of a fault occurring goes up.

In addition, having an even number of members decreases the probability that if there is a network partition then some subset of your nodes will be able to continue running. For example, if you've got a 6 node cluster then it opens up the possibility that a network partition could partition your nodes into 2 3-node partitions. In such a case then neither partition will be able to communicate with a majority of members and your cluster becomes unavailable.

The counter-intuitive conclusion is that, if you have an even-membered cluster then it is actually beneficial (from a high-availability standpoint) to remove one of the members.