CODEWITHSUNDEEP
zerorpc
undefinederrorname
undefinederrorname

Reputation: 78

Concurrent Tasks/Children Processes in ZeroRPC server

I am using a ZeroRPC server to implement a service. Some of the functions of the service are long CPU intensive tasks. Since I don't want the RPC call to block, they must be handled appropriately. Considering the GIL, threading does not seem like an appropriate choice. I've tried multiprocessing but the children processes are unable to connect as an RPC client to another (unrelated) RPC server. See the code below. It is important that I am able to spawn off multiple children processes, every time the spin_off_worker() is called. What is the best way of handling/designing these long-lasting CPU intensive tasks within the ZeroRPC server?

RPC server #1

import zerorpc
import multiprocessing
import time
import threading

class Foo(object):

    def heavy_task(self):
        # some long processing...

        # this RPC call never gets made...
        cli = zerorpc.Client(connect_to="tcp://0.0.0.0:3334", timeout=30, heartbeat=5)
        cli.test()
        cli.close()
        return

    def spin_off_worker(self):
        s = multiprocessing.Process(target=self.heavy_task)
        s.start()
        s.join()

def start():
    s = zerorpc.Server(Foo(), heartbeat=5)
    s.bind("tcp://0.0.0.0:3333")
    s.run()
    return s

svr_proc = multiprocessing.Process(target=start)
svr_proc.start()

RPC Server 2

import zerorpc
import time
class Foo(object):
    def test(self):
        print "TIGR"

s = zerorpc.Server(Foo(), heartbeat=5)
s.bind("tcp://0.0.0.0:3334")
s.run()

Upvotes: 0

Views: 529

Answers (1)

bombela
bombela

Reputation: 633

You solution is in the right direction. It just needs few tweaks:

  • zerorpc requires gevent, and so, runs in a cooperative coroutine environment. Because your worker (server #2) process is going to hold on the CPU for a while, zerorpc will miss heartbeats. You can disable the heartbeat between your master (server #1) and worker (server #2) process. Set heartbeat=None on both side.

  • since you are doing local interprocess communication, you might as well use a local unix socket (if you have access to unix sockets on your system). Use something like unix://workers.sock. This might be more efficient than having the full TCP stack on the way.

Heart-beating is more important on the network than locally on the same machine anyway. If want to run your worker on a different machine across the network, do something like: client machine -> [zerorpc heartbeat] -> server machine -> [local zerorpc no heartbeat] -> worker process.

Note: You do not have to instantiate a new zerorpc client for every heavy_task call. You can reuse the same instance over and over.

Upvotes: 1

Related Questions