number of simultaneous tcp/ip connections on 32 bit linux

Question

I have a specific question on implementing a load balancer or a TCP/IP server program that does TCP/IP.

Since port number is 16 bits, there are a max of only 65536 ports on a single Linux box at any given time. And TCP/IP needs a port number to talk to the outside world. 1) when a client establishes a connection, an ephemeral port number is chosen. 2) when a server listening on a socket accepts a connection, a port number is assigned.

So in my understanding at any given time only maximum 65536 TCP/IP connections can exist on a given machine.

So how is it that some or most load balancers claim 200,000 or more concurrent connections?

Can someone please explain that?

Also regarding load balancers, once a load balancer has forwarded a request to one of the servers behind it, can the load balancer somehow pass some information to it, that will help the server to respond back to the originating client directly to avoid the latency of sending back the response via the load balancer?

Thanks everyone for your help. Thambi

Answer 1

There has been a confusion among this question so I'll try to explain it with examples.

First a couple words about ports: everyone knows that they don't exist physically, they are just an extra identification information for a connection, and also a way to allow multiple servers listening on the same address (if there was no concept of port only one server could be listening on one address, or some other mechanism would have to be in place). Also port is unsigned short so it can have values between 0 and 65535 (64k).

Now, restriction about ports: they are on the server side when bind ing: a (server) socket (let's call it SS) can bind to an address and port: (unless SO_REUSEADDR is set before first binding,) only one socket can listen on a particular address and port at a time, so if someone is already listening on a port you can't listen too. There are some well known ports (e.g.: sshd - 22, httpd - 80, RDP - 3389, ...) that should be avoided when creating SS, a general guidline is never to use a port number < 1k. For a complete list of "reserved" ports, visit www.iana.org.

As stated in the link I posted in the comment there's a 5 item tuple(2 pairs + 1 additional element) that identify a connection (LocalIP: LocalPort, RemoteIP: RemotePort, Protocol) (the last member is just for rigurousity, at this point we don't care about it). Now for a particular SS that listens on a IP:Port, one of the 2 pairs will be the same for all the clients (client sockets: CS) that connect to it depending where looking at the connection from:

• server's endpoint: LocalIP: LocalPort
• client's endpoint: RemoteIP: RemotePort (just like looking in the mirror).

Now I'm going to exemplify on 2 machines (Centos(192.168.149.43) - server and Windows(192.168.137.10) - client). I created a dummy TCP server in Python (note that the code is not structured, no exception handling, only IPv4 capable, the purpose is not to have a Python class but to see some socket behavior):

import sys
import select
import socket

HOST = "192.168.149.43"
PORT = 4461
WAIT_TIME = 0.5

if __name__ == "__main__":
    conns = list()
    nconns = 0
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.bind((HOST, PORT))
    srv.listen(0xFF)
    print "Entering loop, press a key to exit..."
    while sys.stdin not in select.select([sys.stdin], [], [], 0)[0]:
        if select.select([srv], [], [], WAIT_TIME)[0]:
            conn = srv.accept()
            print "Accepted connection from: (%s, %d)" % conn[1]
            conns.append(conn)
            nconns += 1
            print "Active connections:", nconns
    for item in conns:
        item[0].close()
    srv.close()
print "Exiting."

Here's the netstat output on the server machine (before running the server app). I chose port 4461 for communication:

[cfati@xobved-itaf:~]> netstat -an | grep 4461
[cfati@xobved-itaf:~]>

So nothing related to this port. Now after starting the server (I had to trim some spaces so that the output fits here):

[cfati@xobved-itaf:~]> netstat -anp | grep 4461
tcp      0    0 192.168.149.43:4461       0.0.0.0:*             LISTEN

As you can see there is a socket listening for connections on port 4461.

Now going on the client machine and starting the Python interpreter, running the following code in the console:

>>> import sys
>>> import socket
>>> HOST = "192.168.149.43"
>>> PORT = 4461
>>>
>>> def create(no=1):
...     ret = []
...     for i in xrange(no):
...         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
...         s.connect((HOST, PORT))
...         ret.append(s)
...     return ret
...
>>> sockets=[]
>>> sockets.extend(create())

Just after typing the last line on the server machine we look at the server output:

Accepted connection from: (192.168.137.10, 64218)

Active connections: 1

And the corresponding netstat output:

[cfati@xobved-itaf:~]> netstat -an | grep 4461
tcp      0    0 192.168.149.43:4461       0.0.0.0:*             LISTEN
tcp      0    0 192.168.149.43:4461       192.168.137.10:64218  ESTABLISHED

You see the ESTABLISHED connection (this is the accepted socket - AS): The connection was initiated from 192.168.137.10 on port 64218, to 192.168.149.43 on port 4461.

Here's the corresponding netstat output on the client machine (after creating the connection):

e:\Work\Dev>netstat -an | findstr 4461
 TCP    192.168.137.10:64218   192.168.149.43:4461    ESTABLISHED

As you can see the Local and Remote (IP/Port) pairs (compared to the output on the server machine) are reversed (like I mentioned above about looking in the mirror). If I go again on the client machine in the interpreter and re-run the last line (create a new connection):

>>> sockets.extend(create())

the output of the server app will show another entry:

Accepted connection from: (192.168.137.10, 64268)

Active connections: 2

while the netstat output on the server machine:

[cfati@xobved-itaf:~]> netstat -an | grep 4461
tcp      0    0 192.168.149.43:4461       0.0.0.0:*             LISTEN
tcp      0    0 192.168.149.43:4461       192.168.137.10:64268  ESTABLISHED
tcp      0    0 192.168.149.43:4461       192.168.137.10:64218  ESTABLISHED

I'm not posting what netstat will output on the client machine since it's obvious.

Now, let's look at the 2 pairs each corresponding to an active connection: 192.168.137.10:64268, 192.168.137.10:64218. The 2 ports are returned by the accept function (Ux or Win) called on SS.

The 2 ports (64268 and 64218) are used by connections, but that doesn't mean that they cannot be used anymore. Other socket servers can listen on them (I am talking here in the server machine context), or they can be present as used ports in connections from other addresses. Here's a hypothetical netstat output:

tcp      0    0 192.168.149.43:4461       192.168.137.45:64218  ESTABLISHED

So, port 64218 can be also present in a connection from 192.168.137.45 (note that I changed the last IP byte).

As a conclusion, you were somehow right: there can't be more than 65535 (excluding 0 as specified in the other solution) simultaneous connections from the same IP address. This is a huge number, I don't know if in the real world it can be met, but even if so, there are "tricks" to get around it (one example is have 2+ SSs listening on 2+ different ports, and configure the client that if connection to the server to one port fails to use another, so the max simultaneous connections number from the same address can be increased by a factor equal to the number of ports we have servers listening on).

Load balancers handle connections from multiple addresses, so their number can easily grow to hundreds of thousands.

Answer 2

I TCP connection is uniquely identified by a (remote IP address, remote port, remote IP address, remote port) tuple.

For a typical server application, there is only one to three local IP addresses and one or two local ports. For example, a web server might listen on local addresses ::1, ::ffff:93.184.216.34, and 2606:2800:220:1:248:1893:25c8:1946 (possibly via wildcard addresses, but that's irrelevant), and local ports 80 and 443.

For the simple case of a single local address and port that's still 2^{128 + 16} (less a few for special purpose and broadcast addresses), which will be problematic if you wish to communicate with the entire Earth in units of less than 4 million atoms (which might be possible if you converted all matter on Earth into small viruses).

Answer 3

Since port number is 16 bits, there are a max of only 65536 ports on a single Linux box at any given time.

65535 actually, as you can't use port zero.

when a server listening on a socket accepts a connection, a port number is assigned.

No it isn't. The incoming connection uses the same port it connected to. No new port is assigned on accept().

So in my understanding at any given time only maximum 65536 TCP/IP connections can exist on a given machine.

No, see above. The actual limit is determined by kernel and process resources: open FDs, thread stack memory, kernel buffer space, ... Not by the 16-bit port number.