How to build a server that can handle 20.000 concurrenct connections?

Question

It's hard for me to go into exact detail on what the server needs to do (due to NDAs and what not), but it should be sufficient to say that it needs to handle a lightweight binary protocol with many concurrent connected users, ~20.000 is where we have a pretty decent estimate.

Note that clients won't be sending/receiving constantly, but I need to keep the socket open because when the client needs a response we need it as fast as possible and don't have time for the overhead of opening a new connection every time.

The protocol is very lightweight, so there won't be a lot of data going over the wire - the main problem is keeping ~20.000 sockets open at the same time.

(I'm aware that the specifications are a little fuzzy, but I really can't go into more detail)

I have a pretty decent idea what of what I need to do and what type of hardware we need for the server(s) but I figured I'd ask here for existing projects, technologies, languages (Erlang for example), etc. that could assist me in building this.

How can this be achieved?

Answer 1

Maintaining 20,000 connected sockets is not a problem. You can do it using C on Windows (Server) rather easily as long as you use I/O completion ports and/or the threadpool APIs.

The real prblem I guess is generating the data for those 20,000 connections. That might require some exotic solutions - Erlang or whatever. But the socket side of things is not trivial, but well within the bounds of traditional service design.

Answer 2

You don't need to support 20K concurrent users on a single server. Load balance between three or four, and have them connect to the back end database if you're doing any database work; perhaps throw in memcache for good measure, depending on what app you're building.

Answer 3

If you don't have to go through a firewall, consider using a protocol based on UDP. NFS is a good example of a UDP-based protocol. UDP doesn't have the setup overhead of TCP and can scale to more than 65k concurrent connections. However, if you need guaranteed delivery you will have to build this functionality into the application.

For performance with large user bases, consider using a server architecture based on non-blocking I/O.

Another item that might be worth looking at is Douglas Schmidt's Adaptive Communications Environment (ACE). It's a mature C++ framework for building high performance servers, mainly aimed at telecommunications applications. It supports a variety of threading models and handles most of the tricky stuff for you. You might find that the time spent up front learning how to drive it would be saved down the track in reduced debugging effort on messy synchronisation issues.

Answer 4

Take a look at the CCR from the robotics guys at Microsoft. I enables you to do Erlang type programming (message passing, queues, etc), but just using c# and not a totally new functional language.

Furthermore it is able to make use of the asynchronous programming model where you don't need dozens of threads in threadpools to do your stuff. It's much faster and gives really elegant code.

I'm using it myself for an SMS server which needs to spit out SMS's at ridiculous speeds, and it does so without stressing the CPU at all

Answer 5

Erlang with its lightweight threads and awesome binary handling will make it a great fit. As far as the hardware goes, I can't see that you will need an extremely expensive server if the protocol is very lightweight, but that would depend on other processing that needs to be done after the packet have been received.

Edit

If you need to do data lookups by index or something Mnesia is also greater and supports both in memory and disk based storage and is fully distributed if you end up needing to move to more servers

Some real world info on Erlangs connection handling capabilities http://www.sics.se/~joe/apachevsyaws.html