Python closing sockets at random it seems

Question

I've been looking and dealing with this issue for a week. I have client code that causes select() to return a socket that has actually closed from external reasons throwing an error 9 BAD FILE DESCRIPTOR, however I tested the code from a different python file and CANNOT get it to error. Ive tried a million things. heres a snippet from the server:

NOTE: This will work for a few iterations and then suddenly break, it errors out in the message_queue as key error due to the file descriptor breaking even tho a message/no message has a key for that socket present.

#Create the socket to communicate with uWSGI applications
server_address = ('localhost', 10001)
server = create_server_socket(server_address)
#Sockets which we expect to read on from select()
input_sockets = [server]
#Sockets which we expect to write to from select()
output_sockets = []
#Message buffer dicitonary for outgoing messages
message_queue = {}
#Now wait for connections endlessly
while input_sockets:
    print >> sys.stderr, "Waiting for the next event..."
    readable, writable, exceptional = select.select(input_sockets, output_sockets, input_sockets)
    #Handle input_sockets
    for s in readable:
        #Server socket is available for reading now
        if s is server:
            #Create a connection and address object when incoming request is recieved
            connection, client_addr = s.accept()
            print >> sys.stderr, "Connection recieved from %s!" % (client_addr,)
            #Set client connection to non blocking as well
            connection.setblocking(0)
            #Add this socket to input sockets as it will read for client data
            input_sockets.append(connection)
            #Give connection a queue for sending messages to it
            message_queue[connection] = Queue.Queue()
        #A client has sent data so we can handle its request
        else:
            #Pull data from the client
            data = ""
            try:
                while True:
                    message = s.recv(1024)
                    if not message:
                        break
                    data += message
            except Exception as e:
                print str(e)
            if data:
                #Readable client socket has data
                print >> sys.stderr, 'Recieved "%s" from %s' % (data, s.getpeername())
                message_queue[s].put(data)

                #Add output channel now to send message
                if s not in output_sockets:
                    output_sockets.append(s)
            #There is no data to be read, socket must be closed
            else:
                print >> sys.stderr, 'Closing', client_addr,'after recieving no data.'
                #Stop listening for input on the socket
                if s in output_sockets:
                    output_sockets.remove(s)
                input_sockets.remove(s)
                #Close the connection
                s.close()
                del message_queue[s]
    #Handle writable connections    
    for s in writable:
        if s:
            try:
                next_message = message_queue[s].get_nowait()
            except:
                print >> sys.stderr, 'No data to send for', s.getpeername()
                output_sockets.remove(s)
            else:
                try:
                    print >> sys.stderr, 'Sending "%s" to %s' % (next_message, s.getpeername())
                    s.sendall(next_message)
                except:
                    print >> sys.stderr, 'No data to send for', s.getpeername()
                    output_sockets.remove(s)
                #s.sendall('EOF:!@#$:EOF')
    #Now handle any exceptions
    for s in exceptional:
        print >> sys.stderr, 'Handling exception on ', s.getpeername()
        input_sockets.remove(s)
        if s in output_sockets:
            output_sockets.remove(s)
        s.close()
        #Remove any messages
        del message_queue[s]

client:

messages = [ 'This is the message. ',
         'It will be sent ',
         'in parts.',
         ]
server_address = ('localhost', 10001)

# Create a TCP/IP socket
socks = [ socket.socket(socket.AF_INET, socket.SOCK_STREAM),
          socket.socket(socket.AF_INET, socket.SOCK_STREAM),
          ]

# Connect the socket to the port where the server is listening
print >>sys.stderr, 'connecting to %s port %s' % server_address
for s in socks:
    s.connect(server_address)
for message in messages:

    # Send messages on both sockets
    for s in socks:
        print >>sys.stderr, '%s: sending "%s"' % (s.getsockname(), message)
        s.send(message)

    # Read responses on both sockets
    for s in socks:
        data = s.recv(1024)
        print >>sys.stderr, '%s: received "%s"' % (s.getsockname(), data)
        if not data:
            print >>sys.stderr, 'closing socket', s.getsockname()
            s.close()

NOTE: This client side is only to test and start passing messages.

Answer 1

There is a race in your code when a socket is returned as both readable and writable and you close the socket because the read returned 0 bytes. In this case you remove the socket from input_sockets, output_sockets and message_queue but the closed socket is still in writable and it will thus try to write to it inside the same iteration of the select loop.

I have no idea if this is the race you'll see because you neither show debug output not did you say where you stumble over this EBADF. To track similar problems down I recommend to augment your code with more debug information on where you close a socket and where you try to process a socket because it is readable or writable so that you actually find the exact place of the race when looking at the debug output.