CODEWITHSUNDEEP
pythonmultithreadingzerorpc
Manuel S.
Manuel S.

Reputation: 63

python zerorpc multithreading

I'm trying to build a front end service for a python program. I choosed zerorpc to communicate between NodeJS and python, which works fine, the only problem that I do have is that I can't properly multithread zerorpc.
I have seen several post talking about how zerorpc and normal threads don't like each other. However my backend service is already somewhat advanced and uses several threads.
So now the question, is it possible to somehow combine both, normal threads and zerorpc, or do I have to rewrite my main code to utilize the same threading library as zerorpc?
The main problem that I have encountered with normal threads is that the thread that launches the server freezes completely, and therefor can't be closed anymore. Here is how I currently launch the server: 

self.communication_thread = Thread(target=communication_server.start_communication_server)
self.communication_thread.start()

def start_communication_server():
    global server
    addr = 'tcp://127.0.0.1:4242'
    server = zerorpc.Server(CommunicationServer())
    server.bind(addr)
    print('Created a new communicationserver running on {}'.format(addr))
    server.run()

I also tried to kill the server with the following function, which however leads to a exception which is not that desirable

def kill_server():
    global server
    server.stop()

Thanks already in advance for helping me!

Upvotes: 0

Views: 827

Answers (1)

bombela
bombela

Reputation: 633

zerorpc achieves concurrency with gevent (ioloop + coroutines etc). Gevent is single-threaded, and after initialization, can only be used from the thread used to initialize it in the first place. In your case, this is the thread calling server.run(). Therefore, you must only interact with zerorpc methods from the same thread.

You could monkey patch (http://www.gevent.org/intro.html#monkey-patching) which effectively makes your Thread object behave like a coroutine. If you are using threading for IO concurrency only, then this might be good enough.

Otherwise, if you want threads for CPU parallelism, a proven solution is to use sub-process workers as individual threads. You can use zerorpc between your main server and workers (unix sockets are quite a nice fit here). Because workers are CPU bound and local anyway, you can disable the heartbeat between the Server and Workers (Client/Server(heartbeat=0)).

I couldn't find any details about how to safely mix system threads and gevent however.

Upvotes: 1

Related Questions