CODEWITHSUNDEEP
pythonudpmultiprocessing
Serge Tarkovski
Serge Tarkovski

Reputation: 1889

Correct multiprocessing to treat UDP in Python

I am trying to implement a simple UDP client and server. Server should receive a message and return a transformed one.

My main technique for server is to listen UDP messages in a loop, then spawn multiprocessing.Process for each incoming message and send the reply within each Process instance:

class InputProcessor(Process):
    ...

    def run(self):
        output = self.process_input()
        self.sock.sendto(output, self.addr) # send a reply

if __name__ == "__main__":
    print "serving at %s:%s" % (UDP_IP, UDP_PORT)

    sock = socket.socket(socket.AF_INET,    # Internet
                         socket.SOCK_DGRAM) # UDP
    sock.bind((UDP_IP,UDP_PORT))

    while True:
        data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
        print "received message: %s from %s:%s" % (data, addr[0], addr[1])
        p = InputProcessor(sock, data, addr)
        p.start()

In test client, I do something like this:

def send_message(ip, port, data):
    sock = socket.socket(socket.AF_INET,    # Internet
                         socket.SOCK_DGRAM) # UDP
    print "sending: %s" % data
    sock.sendto(data, (ip, port))
    sock.close()


for i in xrange(SECONDS*REQUESTS_PER_SECOND):
    data = generate_data()
    p = multiprocessing.Process(target=send_message, args=(UDP_IP,
                                                           UDP_PORT,
                                                           data))
    p.start()
    time.sleep(1/REQUESTS_PER_SECOND)

The problem I am having with the code above is that when REQUESTS_PER_SECOND becomes higher than certain value (~50), it seems some client processes receive responses destinated to different processes, i.e. process #1 receives response for process #2, and vice versa.

Please criticize my code as much as possible, due to I am new to network programming and may miss something obvious. Maybe it's even worth and better for some reason to use Twisted, hovewer, I am highly interested in understanding the internals. Thanks.

Upvotes: 3

Views: 1613

Answers (2)

jhonkola
jhonkola

Reputation: 3435

As per previous answer, I think that the main reason is that there is a race condition at the UDP port for the clients. I do not see receiving at the client code, but presumably it is similar to the one in server part. What I think happens in concrete terms is that for values under 50 requests / second, the request - response roundtrip gets completed and the client exits. When more requests arrive, there may be multiple processes blocking to read the UDP socket, and then it is probably nondeterministic which client process receives the incoming message. If the network latency is going to be larger in the real setting, this limit will be hit sooner.

Upvotes: 2

Serge Tarkovski
Serge Tarkovski

Reputation: 1889

Thanks guys a lot! It seems I've found why my code failed before. I was using multiprocessing.Manager().dict() within client to check if the results from server are correct. However, I didn't use any locks to wrap a set of write operations to that dict(), thus got a lot of errors though the output from server was correct.

Shortly, in client, I was doing incorrect checks for correct server responses.

Upvotes: 1

Related Questions