CODEWITHSUNDEEP
linuxsocketsconnectionlimit
user3605813
user3605813

Reputation: 11

linux outbound connections timeout or fail after approx 700 established connections

One of our linux servers today experienced problems opening outbound requests. I've reviewed this answer, Increasing the maximum number of tcp/ip connections in linux and it appears as though we are well within the maximum limits.

At the time, netstat -an showed approximately 700 established connections. Any new socket connections would fail, but nothing would be written to /var/log. All connections are long-term, and usually open for several hours at a time.

Is there any logging that would help determine what configuration parameter we are bumping against?

Upvotes: 1

Views: 1805

Answers (1)

xds2000
xds2000

Reputation: 1194

nf_conntrack_tcp_timeout_established

It turns out that there’s another timeout value you need to be concerned with. The established connection timeout. Technically this should only apply to connections that are in the ESTABLISHED state, and a connection should get out of this state when a FIN packet goes through in either direction. This doesn’t appear to happen and I’m not entirely sure why.

So how long do connections stay in this table then? The default value for nf_conntrack_tcp_timeout_established is 432000 seconds. I’ll wait for you to do the long division…

Fun times.

I changed the timeout value to 10 minutes (600 seconds) and in a few days time I noticed conntrack_count go down steadily until it sat at a very manageable level of a few thousand.

We did this by adding another line to the sysctl file:

net.netfilter.nf_conntrack_tcp_timeout_established=600

Upvotes: 1

Related Questions