Redundancy without central control point?

Question

If it possible to provide a service to multiple clients whereby if the server providing this service goes down, another one takes it's place- without some sort of centralised "control" which detects whether the main server has gone down and to redirect the clients to the new server?

Is it possible to do without having a centralised interface/gateway?

In other words, its a bit like asking can you design a node balancer without having a centralised control to direct clients?

Answer 1

This answer is a general overview to high availability for networked applications, not specific to Erlang. I don't know too much about what is available in the OTP framework yet because I am new to the language.

---

There are a few different problems here:

1. Client connection must be moved to the backup machine
2. The session may contain state data
3. How to detect a crash

Problem 1 - Moving client connection
This may be solved in many different ways and on different layers of the network architecture. The easiest thing is to code it right into the client, so that when a connection is lost it reconnects to another machine.

If you need network transparency you may use some technology to sync TCP states between different machines and then reroute all traffic to the new machine, which may be entirely invisible for the client. This is much harder to do than the first suggestion.

I'm sure there are lots of things to do in-between these two.

Problem 2 - State data
You obviously need to transfer the session state from the crashed machine unto the backup machine. This is really hard to do in a reliable way and you may lose the last few transactions because the crashed machine may not be able to send the last state before the crash. You can use a synchronized call in this way to be really sure about not losing state:

1. Transaction/message comes from the client into the main machine.
2. Main machine updates some state.
3. New state is sent to backup machine.
4. Backup machine confirms arrival of the new state.
5. Main machine confirms success to the client.

This may potentially be expensive (or at least not responsive enough) in some scenarios since you depend on the backup machine and the connection to it, including latency, before even confirming anything to the client. To make it perform better you can let the client check with the backup machine upon connection what transactions it received and then resend the lost ones, making it the client's responsibility to queue the work.

Problem 3 - Detecting a crash
This is an interesting problem because a crash is not always well-defined. Did something really crash? Consider a network program that closes the connection between the client and server, but both are still up and connected to the network. Or worse, makes the client disconnect from the server without the server noticing. Here are some questions to think about:

• Should the client connect to the backup machine?
• What if the main server updates some state and send it to the backup machine while the backup have the real client connected - will there be a data race?
• Can both the main and backup machine be up at the same time or do you need to shut down work on one of them and move all sessions?
• Do you need some sort of authority on this matter, some protocol to decide which one is master and which one is slave? Who is that authority? How do you decentralise it?
• What if your nodes loses their connection between them but both continue to work as expected (called network partitioning)?

Answer 2

See Google's paper "Chubby lock server" (PDF) and "Paxos made live" (PDF) to get an idea.

Briefly,this solution involves using a consensus protocol to elect a master among a group of servers that handles all the requests. If the master fails, the protocol is used again to elect the next master.

Also, see gen_leader for an example in leader election which works with detecting failures and transferring service ownership.

Answer 3

Well, you are not giving much information about the "service" you are asking about, so I'll answer in a generic way.

For the first part of my answer, I'll assume you are talking about a "centralized interface/gateway" involving ip addresses. For this, there's CARP (Common Address Redundancy Protocol), quoted from the wiki:

The Common Address Redundancy Protocol or CARP is a protocol which allows multiple hosts on the same local network to share a set of IP addresses. Its primary purpose is to provide failover redundancy, especially when used with firewalls and routers. In some configurations CARP can also provide load balancing functionality. It is a free, non patent-encumbered alternative to Cisco's HSRP. CARP is mostly implemented in BSD operating systems.

Quoting the netbsd's "Introduction to CARP":

CARP works by allowing a group of hosts on the same network segment to share an IP address. This group of hosts is referred to as a "redundancy group". The redundancy group is assigned an IP address that is shared amongst the group members. Within the group, one host is designated the "master" and the rest as "backups". The master host is the one that currently "holds" the shared IP; it responds to any traffic or ARP requests directed towards it. Each host may belong to more than one redundancy group at a time.

This might solve your question at the network level, by having the slaves takeover the ip address in order, without a single point of failure.

Now, for the second part of the answer (the application level), with distributed erlang, you can have several nodes (a cluster) that will give you fault tolerance and redundancy (so you would not use ip addresses here, but "distributed erlang" -a cluster of erlang nodes- instead).

You would have lots of nodes lying around with your Distributed Applciation started, and your application resource file would contain a list of nodes (ordered) where the application can be run.

Distributed erlang will control which of the nodes is "the master" and will automagically start and stop your application in the different nodes, as they go up and down.

Quoting (as less as possible) from http://www.erlang.org/doc/design_principles/distributed_applications.html:

In a distributed system with several Erlang nodes, there may be a need to control applications in a distributed manner. If the node, where a certain application is running, goes down, the application should be restarted at another node.

The application will be started at the first node, specified by the distributed configuration parameter, which is up and running. The application is started as usual.

For distribution of application control to work properly, the nodes where a distributed application may run must contact each other and negotiate where to start the application.

When started, the node will wait for all nodes specified by sync_nodes_mandatory and sync_nodes_optional to come up. When all nodes have come up, or when all mandatory nodes have come up and the time specified by sync_nodes_timeout has elapsed, all applications will be started. If not all mandatory nodes have come up, the node will terminate.

If the node where the application is running goes down, the application is restarted (after the specified timeout) at the first node, specified by the distributed configuration parameter, which is up and running. This is called a failover

distributed = [{Application, [Timeout,] NodeDesc}]

If a node is started, which has higher priority according to distributed, than the node where a distributed application is currently running, the application will be restarted at the new node and stopped at the old node. This is called a takeover.

Ok, that was meant as a general overview, since it can be a long topic :)

For the specific details, it is highly recommended to read the Distributed OTP Applications chapter for learnyousomeerlang (and of course the previous link: http://www.erlang.org/doc/design_principles/distributed_applications.html)

Also, your "service" might depend on other external systems like databases, so you should consider fault tolerance and redundancy there, too. The whole architecture needs to be fault tolerance and distributed for "the service" to work in this way.

Hope it helps!