Determine if peer has closed reading end of socket

Question

I have a socket programming situation where the client shuts down the writing end of the socket to let the server know input is finished (via receiving EOF), but keeps the reading end open to read back a result (one line of text). It would be useful for the server to know that the client has successfully read the result and closed the socket (or at least shut down the reading end). Is there a good way to check/wait for such status?

Answer 1

TL;DR:

Don't rely on the socket state for this; it can cut you in many error cases. You need to bake the acknowledgement/receipt facility into your communications protocol. First character on each line used for status/ack works really well for text-based protocols.

---

On many, but not all, Unix-like/POSIXy systems, one can use the TIOCOUTQ (also SIOCOUTQ) ioctl to determine how much data is left in the outgoing buffer.

For TCP sockets, even if the other end has shut down its write side (and therefore will send no more data to this end), all transmissions are acknowledged. The data in the outgoing buffer is only removed when the acknowledgement from the recipient kernel is received. Thus, when there is no more data in the outgoing buffer, we know that the kernel at the other end has received the data.

Unfortunately, this does not mean that the application has received and processed the data. This same limitation applies to all methods that rely on socket state; this is also the reason why fundamentally, the acknowledgement of receipt/acceptance of the final status line must come from the other application, and cannot be automatically detected.

This, in turn, means that neither end can shut down their sending sides before the very final receipt/acknowledge message. You cannot rely on TCP -- or any other protocols' -- automatic socket state management. You must bake in the critical receipts/acknowledgements into the stream protocol itself.

In OP's case, the stream protocol seems to be simple line-based text. This is quite useful and easy to parse. One robust way to "extend" such a protocol is to reserve the first character of each line for the status code (or alternatively, reserve certain one-character lines as acknowledgements).

For large in-flight binary protocols (i.e., protocols where the sender and receiver are not really in sync), it is useful to label each data frame with an increasing (cyclic) integer, and have the other end respond, occasionally, with an update to let the sender know which frames have been completely processed, and which ones received, and whether additional frames should arrive soon/not-very-soon. This is very useful for network-based appliances that consume a lot of data, with the data provider wishing to be kept updated on the progress and desired data rate (think 3D printers, CNC machines, and so on, where the contents of the data changes the maximum acceptable data rate dynamically).

Answer 2

You're probably better off using ZeroMQ. That will send a whole message, or no message at all. If you set it's send buffer length to 1 (the shortest it will go) you can test to see if the send buffer is full. If not, the message was successfully transferred, probably. ZeroMQ is also really nice if you have an unreliable or intermittent network connection as part of your system.

That's still not entirely satisfactory. You're probably even better off implementing your own send acknowledge mechanism on top of ZeroMQ. That way you have absolute proof that a message was received. You don't have proof that a message was not received (something can go wrong between emitting and receiving the ack, and you cannot solve the Two Generals Problem). But that's the best that can be achieved. What you'll have done then is implement a Communicating Sequential Processes architecture on top of ZeroMQ's Actor Model which is itself implemented on top of TCP streams.. Ultimately it's a bit slower, but your application has more certainty of knowing what's gone on.

Answer 3

Okay so I recall pulling my hair out trying to solve this very problem back in the late 90's. I finally found an obscure doc that stated that a read call to a disconnected socket will return a 0. I use this fact to this day.

Answer 4

getsockopt with TCP_INFO seems the most obvious choice, but it's not cross-platform.

Here's an example for Linux:

import socket
import time
import struct
import pprint


def tcp_info(s):
    rv = dict(zip("""
            state ca_state retransmits probes backoff options snd_rcv_wscale
            rto ato snd_mss rcv_mss unacked sacked lost retrans fackets
            last_data_sent last_ack_sent last_data_recv last_ack_recv
            pmtu rcv_ssthresh rtt rttvar snd_ssthresh snd_cwnd advmss reordering
            rcv_rtt rcv_space
            total_retrans
            pacing_rate max_pacing_rate bytes_acked bytes_received segs_out segs_in
            notsent_bytes min_rtt data_segs_in data_segs_out""".split(),
                  struct.unpack("BBBBBBBIIIIIIIIIIIIIIIIIIIIIIIILLLLIIIIII",
                                s.getsockopt(socket.IPPROTO_TCP, socket.TCP_INFO, 160))))
    wscale = rv.pop("snd_rcv_wscale")

    # bit field layout is up to compiler
    # FIXME test the order of nibbles
    rv["snd_wscale"] = wscale >> 4
    rv["rcv_wscale"] = wscale & 0xf
    return rv

for i in range(100):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(("localhost", 7878))
    s.recv(10)
    pprint.pprint(tcp_info(s))

I doubt a true cross-platform alternative exists.

Fundamentally there are quite a few states:

• you wrote data to socket, but it was not sent yet
• data was sent, but not received
• data was sent and losts (relies on timer)
• data was received, but not acknowledged yet
• acknowledgement not received yet
• acknowledgement lost (relies on timer)
• data was received by remote host but not read out by application
• data was read out by application, but socket still alive
• data was read out, and app crashed
• data was read out, and app closed the socket
• data was read out, and app called shutdown(WR) (almost same as closed)
• FIN was not sent by remote yet
• FIN was sent by remote but not received yet
• FIN was sent and got lost
• FIN received by your end

Obviously your OS can distinguish quite a few of these states, but not all of them. I can't think of an API that would be this verbose...

Some systems allow you to query remaining send buffer space. Perhaps if you did, and socket was already shut down, you'd get a neat error?

Good news is just because socket is shut down, doesn't mean you can't interrogate it. I can get all of TCP_INFO after shutdown, with state=7 (closed). In some cases report state=8 (close wait).

http://lxr.free-electrons.com/source/net/ipv4/tcp.c#L1961 has all the gory details of Linux TCP state machine.

Answer 5

No. All you can know is whether your sends succeeded, and some of them will succeed even after the peer read shutdown, because of TCP buffering.

This is poor design. If the server needs to know that the client received the data, the client needs to acknowledge it, which means it can't shutdown its write end. The client should:

1. send an in-band termination message, as data.
2. read and acknowledge all further responses until end of stream occurs.
3. close the socket.

The server should detect the in-band termination message and:

1. stop reading requests from the socket
2. send all outstanding responses and read the acknowledgements
3. close the socket.

OR, if the objective is only to ensure that client and server end at the same time, each end should shutdown its socket for output and then read input until end of stream occurs, then close the socket. That way the final closes will occur more or less simultaneously on both ends.