How many long-living concurrent TCP sessions are consideer reasonable?

Question

I am developing a TCP server in Linux which initially can handle thousands of concurrent clients, which are intendeed to be long living. However, after starting to implement some functionallity, I made a thread pool for that calls which are blocking and should be done apart, like database or disk access.

After some tests, under high load requesting "many" asynchronous functions my server starts to lag due to many tasks being enqueued, as they arrive faster than they can be processed. These tasks are solved in the nanoseconds, but there are thounsands. I do understand this is totally normal.

I could of course grow behind a load balancer or buying better servers with more cores, however, in practice and as standard in the industry, how many concurrent long-lived TCP sessions are consideer a "good" number in such a server like this one I'm describing? How can I say that the number of concurrent connections I got is "good enough"?

Answer 1

Unfortunately there's no a magic number to answer your question but I have some considerations for you to find your number:

• First of all, every operational system has its own max number of simultaneous connections because the port numbers are finite. So check if you're not trespassing this number, else every new connection will be refused by your server.
• In order to identify how many simultaneous connections are okay to you, you must to establish a max time of response for your service.

Keep in mind that even having simultaneous connections, multicore CPUs etc... the response will come out by the same network and get bottle necked. Thus I advice you to do a load and a stress test over your architecture in order to find your acceptable latency limit.

---

TL;DR: There's no a magic answer, you should do a load and a stress test to find it.