Synchronise a variable between java instance across network

Question

I have this assignment in college where they ask us to run a Java app as a socket server with multiple clients. Client sends a string, server returns the string in upper case with a request counter. Quite simple.

Each request made by any given client is counted on the server side and stored in a static variable for each client connection thread. So that each client request increments the counter globally on the server. That's working well.

Now, they ask us to run "backup" instances of that server on different machines on the network so that if the primary stops responding, the client connects to one of the backups. That, I got working. But the counter is obviously reset since it's a different server.

The challenge is that the request counter be the same on the primary and the secondaries so that if the primary responds to 10 requests, goes down, client switch to a backup and makes a request, the backup server responds 11.

Here is what I considered:

1. if on the same PC, I'd use threads but we're over the network so I believe this will not work.
2. server sends that counter to the client with the response, which in turn returns it to the server at the next request and so forth. Not very "clean" imo but could work.
3. Each server talks to each other to sync this counter. However, sockets don't seem to be very efficient to do this, if even possible. RMI seems to be an alternative here but I'd like confirmation before I start learning it.

Any leads or suggestions here? I'm not posting code because I don't need a solution here but if necessary, I can invite to the gihub repo.

EDIT: There is no latency, reliability or similar constraints for this project. There is X number of clients and Y number of servers (single master, multiple failovers). Additional third party infrastructure like a DB isn't an option really but third party Java librairies are welcome. Basically I just run in Eclipse on multiple PCs. This is an introduction assignment to distributed systems, expected done in 2 weeks so I believe "keep it simple" is the key here!

EDIT 2: The number and addresses of backup servers will be passed as arguments to the application so broadcast/discovery isn't necessary. We'll likely cover all those points in a later lab assignment in the semester :)

EDIT 3: From all your great suggestions, I'll try an implementation of some variation of #3 and let you know how it works. I think the issue I have here is to make sure all servers are aware of the others. But like I mentioned, they don't need to discover each other so I'll hard code it for now and revisit in the next assignment! Probably opt for some elected master... :)

Answer 1

In my opinion best solution is to have vector of the counters. One counter per one server. Each server increments its own counter and broadcast vector value to all other servers. This data structure is conflict-free replicated data type.

Number of requests is calculated as sum of all elements of the vector.

About consistency. If you need strictly growing number on all servers you need to synchronously replicate you new value before answer to client. The penalty here is performance and availability.

About broadcasting. You can choose any broadcasting algorithm you want. If number of servers are not too large you can use full mesh topology. If number of server become large you can use ring or star topology to replicate data.

Answer 2

Solutions to this problem trade off speed against consistency.

If you value consistency over speed you could try a synchronous approach (assuming servers A, B and C):

1. A receives initial request
2. A opens connection to B and C to request current counts from each
3. A calculates max count (based on its own value and the values from B and C), adds one and sends new count to B and C
4. A closes connections to B and C
5. A replies to original request, including new max count

At this point, all servers are in sync with the new max count, ready for a new request to any server.

Edit: Of course, if you are able to use a shared database, the problem becomes much simpler.

Answer 3

This is a classic distributed systems problem. The right solution is some variation of your option #3, where the different servers communicate with each other.

Where it gets complicated is when you start to introduce latency, downtime, and/or network partitioning between the various servers. Eventually you'll need to arrive at some kind of consensus algorithm. Paxos is a well-known approach to this problem, but there are others; Raft is popular these days as well.

Answer 4

If option #2 is allowed, then it is the easiest, however I am not sure how it could work in the face of multiple clients (so it depends on the requirements here).

Is it possible to back the servers by a shared db running on another computer? Ideally perhaps one clustered across multiple machines? Or can you use an event bus or 3rd party libraries / code (shared cache, JMS, or even EJBs)?

If not, then having the servers talk to each other is your best bet. Sockets can work, as could UDP multicast (careful there though, no way to know if a message was missed which is why TCP / sockets are safer). If the nodes are going to talk to each other there are generally a few accepted ways to handle the setup:

• Master / slaves: Current node is the master and all writes are to it. Slaves connect to the master and receive updates. When the master goes down a new master needs to be elected (see leader election). MongoDB works like this.
• Everyone to everyone: Every node connects to every other known node. Can get complicated and might not scale well to lots of nodes.
• Daisy chain: one node connects to the next node, which connects to the next, and so on. I don't believe this is widely used.
• Ring network: Each node connects to two others in order to form a ring. This is generally superior to daisy chain, but a little bit more complicated to implement.

See here for more examples: https://en.wikipedia.org/wiki/Network_topology

If this was in the real world (i.e. not school), you would use either a shared cache (e.g. ehcache), local caches backed by an event bus (JMS of some sort), or a shared clustered db.

---

EDIT:

After re-reading your question, it seems you only have a single backup server to worry about, and my guess of the course requirements is that they simply want your backup server to connect to your primary server and also receive the variable count updates. This is completely fine to implement with sockets (it isn't inefficient for a single backup server), and is perhaps the solution they are expecting you to use.

E.g. Backup server connects to primary server and either polls for updates across the held connection or simply listens for updates issued from the primary server.

Key notes: - You might need keep alives to ensure the connection does not get killed. - Make sure to implement re-connection logic if the connection from backup to primary dies.

If this is for a networking course they may be expecting UDP multicast, but that may depend a little bit on the server / network environment.

Answer 5

The most real life would be option 3. It happens all the time. Nodes talk to one another on another port. So they self discover by broadcast (UDP). So each server broad casts its max on a UDP port. Other nodes listen and up their value that + 1 if their current value is less than that value, else ignore it and instead broadcast their bigger value.

This will work best when there is a 2-300 gap between client calls. This also assumes that any server could be primary (as decided by a load balancer).

UDP is stable within a LAN. Used widely.