How do I manage metainformation in a horizontally scaled game server?

Question

I apologize in advance for how long this explanation is, I don't know how to make it more concise because I imagine almost all of this is relevant. Sorry!

I'm designing a game server in Python with Twisted (probably not relevant, as this is an architecture question).

The general idea is that players connect and are placed into a lobby. They can then queue for a match (for instance, deathmatch or team deathmatch), and, when a match is found, they are placed into that game. When the game ends they are placed back into the lobby.

Because I'm aware of how complex distributed systems can be, I tried to simplify it as much as possible. The idea I came up with was to abstract all the information about a player into an object. All game logic is implemented in a GameHandler. When a player connects, they're assigned to a Lobby(GameHandler) instance. When they join a match, they are reassigned to a, say, Deathmatch(GameHandler) instance (which are held in a map of server: gamehandler).

At that point, when the player is added to a match, the server they're actually connected to serves as a reverse proxy (I didn't write the client and it can't be modified, there can't be any connection re-negotiation) and sends the info about the player to the match server. Then, using the instance map, all traffic from that player is routed without being looked at to the game instance they're in, and vice versa with an ID system. I don't think this is a problem because the servers should all be able to forward data on a gigabit LAN.

When a game is over, it just notifies all the Lobby servers (reverse proxies) that forwarded players, and they're returned back to the Lobby instance.

That should mean that I can scale out with resources by adding backend servers, scale out with network links by adding more reverse proxy lobby-only servers, and I can also scale up on any of the individual nodes. There is also no technical limitation that forces backend servers to be backend, every single server could have a Lobby instance, but games could be distributed across the network.

Now, so far so good (In theory, I haven't started implementing the distribution yet because I want to work out the main points beforehand), but that leaves one major question:

How do I control metainformation that all nodes need to know?

For instance:

• How many players are online for the server list to display before a client connects?
• Is matchmaking implemented (I'm planning on using Microsoft's TrueSkill algorithm, if that matters) in some P2P manner or should I delegate an entire server just for that (or even for metainformation)?
• What about a party system where players join a queue together? Which server keeps track of the players in the group?
• How do I manage configuration, like banned players? Every forward server would need to know about them.
• If the lobby instance the player connected to happens to be full, how do I find another lobby instance that isn't full so I can route their connection there? This goes back to the first point, I need to have a way for nodes to easily query the network state

I could implement a P2P solution, or a primary server to handle this sort of thing. My major concern would be that adding a primary control server would add a single point of failure (which I would like to avoid), but the P2P solution would seem to be an order of magnitude more complex, and potentially slow things down significantly or use a fair amount of resources caching data on all the nodes.

So the question is: Is my design decent, and what do you think the pros and cons of those two solutions to the metainformation problem are? Is there a better third solution? What do you think is best?

Thank you in advance, your help is very much appreciated.

Answer 1

There are many solutions to implement a shared database. It depends on your technology stack, network architecture, programming language(s), etc. This is too broad of a question to be answered in a few paragraphs. Pick one approach and go with it, but make your code modular enough to replace your approach with another if necessary.

Update: Based on your comment that you are using Twisted, I will ask you a question. If you had a cluster of Twisted servers that are all sharing network state (your "distributed nodes"), how would you request your "complex operations" from those servers and how would you get back the results? If you can answer this in enough detail, you will have determined the requirements of your nodes. And then you can determine how they share and update the network state. At that point, you can ask a much more specific question (like "how do I replicate memcache across my nodes?").