CODEWITHSUNDEEP
pythonjsonpython-3.xmultithreadingsockets
stevejj
stevejj

Reputation: 3

Is there a way to have multiple threads listen to the same socket in python?

I have a program that takes encoded commands from a host through a socket, using a thread that continuously listens to the socket and starts new threads to handle each command. A command is either a bad message which can be ignored, or it starts with 3 letter indicating what to do, then a %, then content in json format and ends with \0. Content range in length from 12 to 100000+ chars.

def listenthread():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((TCP_IP, TCP_PORT))
    while True:
        while True:
            command_byte = b""
            current_byte = sock.recv(1)
            if current_byte == b"\0":
                break
            command_byte += current_byte
        try:
            command = command_byte.decode()
        except UnicodeDecodeError:
            continue
        threading.Thread(target=handler, args=(command,sock)).start()

The handler calls the appropriate function to handle the content.

def handler(command, sock):
    content = json.loads(command[4:])
    command = command[0:3]
    if command == "mpg":
        func1(content, sock)
    elif command == "qrc":
        func2(content, sock)

Functions can be straight forward, like func1

def func1(content, sock):
    # do stuff
    msg = json.dumps({"succes": True})
    sock.send(f"%{msg}\0".encode())

These work absolutely fine. My problem happens when a function needs to talk to the host again and needs to get the response.

def func2(content, sock):
    # do stuff
    msg = json.dumps({"succes": False, "required": "somestuff"})
    sock.send(f"%{msg}\0".encode())
    while True:
        command_byte = b""
        current_byte = sock.recv(1)
        if current_byte == b"\0":
            break
        command_byte += current_byte
    command = command_byte.decode()
    
    # do more stuff
    msg = json.dumps({"succes": True})
    sock.send(f"%{msg}\0".encode())

The goal is that func2() can request and receive data through the same socket as listenthread() listens too. I've tried implementing this with a Lock where listenthread would temporarily stop listening, but unfortunately random commands can still occur between the requesting and the receiving of data. I have also looked into async coroutines, but have not been able to fix it with those.

Upvotes: 0

Views: 994

Answers (1)

Jeremy Friesner
Jeremy Friesner

Reputation: 73081

Having multiple threads simultaneously use the same TCP socket is possible in that most operating systems support it (i.e. you won't crash), but it's not useful because it's unpredictable which thread will end up receiving which bytes of data from the socket (or conversely, which thread will end up sending which bytes of data to the socket when) -- in either case your TCP data will end up getting randomly send to your various threads, which is a big problem if your messages are ever longer than one byte long.

What you can do, OTOH, is have one thread that is dedicated to only reading from the TCP socket, and when it has read in a full message, it hands that message over to one of the other threads (using a mutex-protected FIFO queue and a condition-variable, or similar) for processing. If the processing-thread later wants to send data back to the TCP socket, it would hand the data back to the network-thread via another FIFO queue, and then the network thread would take care of pulling the data out of the FIFO and writing it out to the TCP socket. In that design, because only one thread has access to the TCP socket, you can guarantee that messages boundaries are respected and messages do not get mixed up with each other.

Note that in order to have the network thread handle both reading and writing (and responding to incoming messages on its FIFO queue) in a timely manner, you may want to have it use non-blocking I/O and event-loop that blocks only inside select().

Upvotes: 1

Related Questions