CODEWITHSUNDEEP
linuxsockets
Jonesome Reinstate Monica
Jonesome Reinstate Monica

Reputation: 6934

Linux: TCP Socket listening: How to detect clients that fail to close socket?

We have an app on linux (CentOS) that dies from time to time. All it does is listen on a socket (and do something with the data coming in).

We are starting to think that some of the clients connecting to to this socket may be failing to close the socket connection, and that this causes the linux app to run out of resources. (The server ultimately throws a "too many open files" error in a netty exception.)

How can we measure this on the server? (We do not have the source code to the server app, but we do have the source to the client.)

netstat -nato seems close, but we are not sure that is the whole story.

Upvotes: 0

Views: 631

Answers (2)

Jonesome Reinstate Monica
Jonesome Reinstate Monica

Reputation: 6934

The solution was to use the ls command, as shown:

 ls -al /proc/<PID>/fd

Using this with the wc counting utility

 ls -al /proc/<PID>/fd | wc -l

Gives us a simple count of the # of resources (pipes, sockets, files) held open by a process.

We then could see that our client code needed a small adjustment to reliably close the socket.

Upvotes: -1

user207421
user207421

Reputation: 310840

Look for ports in CLOSE_WAIT state in the netstat display. If you see lots, you have a resource leak in the server, which is not due to the clients. Conversely if you have lots of ports in FIN_WAIT_1 your server is closing but your clients aren't. Neither of these is exactly what you're looking for but they expose cognate bugs.

If possible, set a read timeout on the accepted sockets, and log and close them if you get it. Choose the timeout interval wisely.

Upvotes: 2

Related Questions