CODEWITHSUNDEEP
python-3.xmultiprocessingzeromqpyzmq
kernstock
kernstock

Reputation: 234

ZeroMQ hangs in a python multiprocessing class/object solution

I'm trying to use ZeroMQ in Python (pyzmq) together with multiprocessing. As a minmal (not) working example I have a server- and a client-class which both inherit from multiprocessing.Process. The client as a child-process should send a message to the server-child-process which should print the message:

#mpzmq_class.py

from multiprocessing import Process
import zmq


class Server(Process):
    def __init__(self):
        super(Server, self).__init__()
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.PULL)
        self.socket.connect("tcp://localhost:6068")

    def run(self):
        msg = self.socket.recv_string()
        print(msg)


class Client(Process):
    def __init__(self):
        super(Client, self).__init__()
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.PUSH)
        self.socket.bind("tcp://*:6068")

    def run(self):
        msg = "Hello World!"
        self.socket.send_string(msg)

if __name__ == "__main__":
    s = Server()
    c = Client()
    s.start()
    c.start()
    s.join()
    c.join()

Now if I run this the server-process seems to hang at the receive-call msg = socket.receive_string(). In another (more complicated) case, it even hung at the socket.connect("...")-statement.

If I rewrite the script to use functions instead of classes/objects, it runs just fine:

# mpzmq_function.py

from multiprocessing import Process
import zmq


def server():
    ctx = zmq.Context()
    socket = ctx.socket(zmq.PULL)
    socket.connect("tcp://localhost:6068")
    msg = socket.recv_string()
    print(msg)


def client():
    ctx = zmq.Context()
    socket = ctx.socket(zmq.PUSH)
    socket.bind("tcp://*:6068")
    msg = "Hello World!"
    socket.send_string(msg)

if __name__ == "__main__":
    s = Process(target=server)
    c = Process(target=client)
    s.start()
    c.start()
    s.join()
    c.join()

Output:

paul@AP-X:~$ python3 mpzmq_function.py 
Hello World!

Can anybody help me with this? I guess it's something I didn't understand concerning the usage of multiprocessing.

Thank you!

Upvotes: 6

Views: 2775

Answers (1)

DeaD_EyE
DeaD_EyE

Reputation: 443

I run into the same issue. I guess the problem is, that the run method has no access to the context object. Maybe it has something to do with the C implementation and the fact, that processes do not have shared memory. If instantiate the context in the run method, it works.

Here a working example:

#mpzmq_class.py

from multiprocessing import Process
import zmq


class Base(Process):
    """
    Inherit from Process and
    holds the zmq address.
    """
    def __init__(self, address):
        super().__init__()
        self.address = address


class Server(Base):
    def run(self):
        ctx = zmq.Context()
        socket = ctx.socket(zmq.PULL)
        socket.connect(self.address)
        msg = socket.recv_string()
        print(msg)


class Client(Base):
    def run(self):
        ctx = zmq.Context()
        socket = ctx.socket(zmq.PUSH)
        socket.bind(self.address)
        msg = "Hello World!"
        socket.send_string(msg)


if __name__ == "__main__":
    server_addr = "tcp://127.0.1:6068"
    client_addr = "tcp://*:6068"
    s = Server(server_addr)
    c = Client(client_addr)
    s.start()
    c.start()
    s.join()
    c.join()

I added a base class to demonstrate that you can still access normal Python objects from the run method. If you put the context object into the init Method, it won't work.

Upvotes: 8

Related Questions