CODEWITHSUNDEEP
linuxnetwork-programmingtcp
Michael Marsee
Michael Marsee

Reputation: 1105

tcp_tw_reuse vs tcp_tw_recycle : Which to use (or both)?

I have a website and application which use a significant number of connections. It normally has about 3,000 connections statically open, and can receive anywhere from 5,000 to 50,000 connection attempts in a few seconds time frame.

I have had the problem of running out of local ports to open new connections due to TIME_WAIT status sockets. Even with tcp_fin_timeout set to a low value (1-5), this seemed to just be causing too much overhead/slowdown, and it would still occasionally be unable to open a new socket.

I've looked at tcp_tw_reuse and tcp_tw_recycle, but I am not sure which of these would be the preferred choice, or if using both of them is an option.

Upvotes: 73

Views: 131272

Answers (4)

Cédric Julien
Cédric Julien

Reputation: 80831

According to Linux documentation, you should use TCP_TW_REUSE flag to allow reusing sockets in TIME_WAIT state for new connections.

It seems to be a good option when dealing with a web server that have to handle many short TCP connections left in TIME_WAIT state.

As described here, TCP_TW_RECYCLE could cause some problems when using load balancers...

EDIT (to add some warnings ;) ):

as mentionned in comment by @raittes, the problems when using load balancers is about public-facing servers. When recycle is enabled, the server can't distinguish new incoming connections from different clients behind the same NAT device.

Upvotes: 72

呂學洲
呂學洲

Reputation: 1423

According to the VMWare document, the main difference is TCP_TW_REUSE works only on outbound communications.

  • TCP_TW_REUSE uses server-side time-stamps to allow the server to use a time-wait socket port number for outbound communications once the time-stamp is larger than the last received packet. The use of these time-stamps allows duplicate packets or delayed packets from the old connection to be discarded safely.

  • TCP_TW_RECYCLE uses the same server-side time-stamps, however it affects both inbound and outbound connections. This is useful when the server is the first party to initiate connection closure. This allows a new client inbound connection from the source IP to the server. Due to this difference, it causes issues where client devices are behind NAT devices, as multiple devices attempting to contact the server may be unable to establish a connection until the Time-Wait state has aged out in its entirety.

Upvotes: 0

pevik
pevik

Reputation: 4801

NOTE: net.ipv4.tcp_tw_recycle has been removed from Linux in 4.12 (4396e46187ca tcp: remove tcp_tw_recycle).

SOURCE: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux

Upvotes: 44

Bernard Rosset
Bernard Rosset

Reputation: 4763

pevik mentioned an interesting blog post going the extra mile in describing all available options at the time.

Modifying kernel options must be seen as a last-resort option, and shall generally be avoided unless you know what you are doing... if that were the case you would not be asking for help over here. Hence, I would advise against doing that.

The most suitable piece of advice I can provide is pointing out the part describing what a network connection is: quadruplets (client address, client port, server address, server port).

If you can make the available ports pool bigger, you will be able to accept more concurrent connections:

  • Client address & client ports you cannot multiply (out of your control)
  • Server ports: you can only change by tweaking a kernel parameter: less critical than changing TCP buckets or reuse, if you know how much ports you need to leave available for other processes on your system
  • Server addresses: adding addresses to your host and balancing traffic on them:
    • behind L4 systems already sized for your load or directly
    • resolving your domain name to multiple IP addresses (and hoping the load will be shared across addresses through DNS for instance)

Upvotes: 0

Related Questions