Python sockets sending unexpected RST packets

Question

I have threaded Python 2.7.12 code that waits for connections from external devices, and then spins off a thread to work with each device. 80% of the time this works fine, but every once in a while I get unwanted RST from the OS or Python, I'm not sure which.

The TCP connection starts off normally and then abrubptly is reset.

SYN, SYN/ACK, ACK, RST

The server initiates the RST. The timing is also extremely tight so I feel like this is probably Python's socket bugging out or the OS taking over for some reason. How could I debug this? Since I'm not seeing any errors in my Python code and logging itself, is there a ways to debug the sockets code directly used by Python?

Here is the code:

while True:
host = socket.gethostbyname(socket.gethostname())
port = 3001
try:
    tcpsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    tcpsock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    tcpsock.bind((host,port))
    tcpsock.listen(5)
    print ("\nListening for incoming connections...")
    (clientsock, (ip, port)) = tcpsock.accept()
    newthread = ClientThread(ip, port, clientsock)
    newthread.start()
    threads = 0
    for t in threading.enumerate():
        threads += 1
    logger.info('######################## THREADS = %s' % (threads))
except Exception as e:
    logger.critical('Exception: %s.  Error initializing thread' % (e))

EDIT1 - Adding ClientThread code:

class ClientThread(threading.Thread):

def __init__(self,ip,port,clientsocket):
    threading.Thread.__init__(self)
    self.ip = ip
    self.port = port
    self.csocket = clientsocket

The rest of the code just starts working with the device.

Answer 1

The RST packet means "I don't have a connection open for the packet you just sent me". You're right, if you follow the simple examples on python.org, then Python will generate RST packets. For example:

#!/usr/bin/python2.7
import socket, sys, time

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

# Bind the socket to the port
server_port = 10000
print 'starting up on port %s' % server_port
sock.bind(('', server_port))

# Listen for incoming connections
sock.listen(1)

print 'waiting for a connection'
conn, cli_addr = sock.accept()
print 'connection from', cli_addr
data = "Hello, World!\n" \
    + "Your IP is " + cli_addr[0] + "\n"
HTTP = "HTTP/1.1 200 OK\nContent-Type: text/html\nContent-Length:"
conn.sendall(HTTP + str(len(data)) + "\n\n" + data);
conn.close()

tcpdump then says:

10:59:48.847443 IP 127.0.0.1.38330 > 127.0.0.1.10000: Flags [S], seq 1326297161, win 43690, options [mss 65495,sackOK,TS val 3126487416 ecr 0,nop,wscale 7], length 0
10:59:48.847462 IP 127.0.0.1.10000 > 127.0.0.1.38330: Flags [S.], seq 1677934801, ack 1326297162, win 43690, options [mss 65495,sackOK,TS val 3126487416 ecr 3126487416,nop,wscale 7], length 0
10:59:48.847475 IP 127.0.0.1.38330 > 127.0.0.1.10000: Flags [.], ack 1, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 0
10:59:48.847547 IP 127.0.0.1.38330 > 127.0.0.1.10000: Flags [P.], seq 1:80, ack 1, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 79
10:59:48.847557 IP 127.0.0.1.10000 > 127.0.0.1.38330: Flags [.], ack 80, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 0
10:59:48.847767 IP 127.0.0.1.10000 > 127.0.0.1.38330: Flags [P.], seq 1:95, ack 80, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 94
10:59:48.847790 IP 127.0.0.1.38330 > 127.0.0.1.10000: Flags [.], ack 95, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 0
10:59:48.847861 IP 127.0.0.1.10000 > 127.0.0.1.38330: Flags [R.], seq 95, ack 80, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 0
10:59:48.847862 IP 127.0.0.1.38330 > 127.0.0.1.10000: Flags [F.], seq 80, ack 95, win 342, options [nop,nop,TS val 3126487416 ecr 3126487416], length 0
10:59:48.847878 IP 127.0.0.1.10000 > 127.0.0.1.38330: Flags [R], seq 1677934896, win 0, length 0

The problem is that conn.close() destroys the socket data structure, so the OS doesn't recognize the next packet. Notice that the client (curl) is sending a nice FIN packet, but the server (Python code above) is not.

To solve this, you need to send a FIN packet by calling conn.shutdown(socket.SHUT_WR) and then leave the socket structure around long enough to catch any straggling packets (you could do this in another thread). So instead of conn.close(), do:

conn.shutdown(socket.SHUT_WR)
time.sleep(1)
conn.close()

With this change, both sides send FIN, and the RST are missing from tcpdump:

11:05:50.201352 IP 127.0.0.1.38338 > 127.0.0.1.10000: Flags [S], seq 4249130338, win 43690, options [mss 65495,sackOK,TS val 3126848770 ecr 0,nop,wscale 7], length 0
11:05:50.201368 IP 127.0.0.1.10000 > 127.0.0.1.38338: Flags [S.], seq 1410528158, ack 4249130339, win 43690, options [mss 65495,sackOK,TS val 3126848770 ecr 3126848770,nop,wscale 7], length 0
11:05:50.201381 IP 127.0.0.1.38338 > 127.0.0.1.10000: Flags [.], ack 1, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0
11:05:50.201441 IP 127.0.0.1.38338 > 127.0.0.1.10000: Flags [P.], seq 1:80, ack 1, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 79
11:05:50.201450 IP 127.0.0.1.10000 > 127.0.0.1.38338: Flags [.], ack 80, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0
11:05:50.201568 IP 127.0.0.1.10000 > 127.0.0.1.38338: Flags [P.], seq 1:95, ack 80, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 94
11:05:50.201594 IP 127.0.0.1.38338 > 127.0.0.1.10000: Flags [.], ack 95, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0
11:05:50.201643 IP 127.0.0.1.10000 > 127.0.0.1.38338: Flags [F.], seq 95, ack 80, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0
11:05:50.201691 IP 127.0.0.1.38338 > 127.0.0.1.10000: Flags [F.], seq 80, ack 96, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0
11:05:50.201707 IP 127.0.0.1.10000 > 127.0.0.1.38338: Flags [.], ack 81, win 342, options [nop,nop,TS val 3126848770 ecr 3126848770], length 0