Can etcd detect problems and elect leaders for other clusters?

Question

To my knowledge, etcd uses Raft as a consensus and leader selection algorithm to maintain a leader that is in charge of keeping the ensemble of etcd nodes in sync with data changes within the etcd cluster. Among other things, this allows etcd to recover from node failures in the cluster where etcd runs.

But what about etcd managing other clusters, i.e. clusters other than the one where etcd runs?

For example, say we have an etcd cluster and separately, a DB (e.g. MySQL or Redis) cluster comprised of master (read and write) node/s and (read-only) replicas. Can etcd manage node roles for this other cluster?

More specifically:

• Can etcd elect a leader for clusters other than the one running etcd and make that information available to other clusters and nodes?

• To make this more concrete, using the example above, say a master node in the MySQL DB cluster mentioned in the above example goes down. Note again, that the master and replicas for the MySQL DB are running on a different cluster from the nodes running and hosting etcd data.

  • Does etcd provide capabilities to detect this type of node failures on clusters other than etcd's automatically? If yes, how is this done in practice? (e.g. MySQL DB or any other cluster where nodes can take on different roles).

  • After detecting such failure, can etcd re-arrange node roles in this separate cluster (i.e. designate new master and replicas), and would it use the Raft leader selection algorithm for this as well?

  • Once it has done so, can etcd also notify client (application) nodes that depend on this DB and configuration accordingly?

  • Finally, does any of the above require Kubernetes? Or can etcd manage external clusters all by its own?

---

In case it helps, here's a similar question for Zookeper.

Answer 1

etcd's master election is strictly for electing a leader for etcd itself.

Other clusters, however can use a distributed strongly-consistent key-value store (such as etcd) to implement their own failure detection, leader election and to allow clients of that cluster to respond.

Etcd doesn't manage clusters other than its own. It's not magic awesome sauce.

If you want to use etcd to manage a mysql cluster, you will need a component which manages the mysql nodes and stores cluster state in etcd. Clients can watch for changes in that cluster state and adjust.