CODEWITHSUNDEEP
pythonnode.jsrpcdistributed
Will
Will

Reputation: 4631

Distributed system: Python 3 worker and Node.js server?

I'm looking to set up a distributed system where there are compute/worker machines running resource-heavy Python 3 code, and there is a single web server that serves the results of the Python computation to clients. I would very much like to write the web server in Node.js.

I've looked into using an RPC framework—specifically, this question lead me to ZeroRPC, but it's not compatible with Python 3 (the main issue is that it requires gevent, which isn't that close to a Python 3 version yet). There doesn't seem to be another viable option for Python–Node.js RPC as far as I can tell.

In light of that, I'm open to using something other than RPC, especially since I've read that the RPC strategy hides too much from the programmer.

I'm also open to using a different language for the web server if that really makes more sense; for example, it may be much simpler from a development point of view to just use Python for the server too.

How can I achieve this?

Upvotes: 0

Views: 743

Answers (2)

abarnert
abarnert

Reputation: 365945

You have a few options here.

First, it sounds like you like ZeroRPC, and your only problem is that it depends on gevent, which is not 3.x-ready yet.

Well, gevent is close to 3.x-ready. There are a few forks of it that people are testing and even using, which you can see on issue #38. As of mid-September 2014 the one that seems to be getting the most traction is Michal Mazurek's. If you're lucky, you can just do this:

pip3 install git+https://github.com/MichalMazurek/gevent
pip3 install ZeroRPC

Or, if ZeroRPC has metadata that says it's Python 2-only, you can install it from its repo the same way as gevent.

The down side is that none of the gevent-3.x forks are quite battle-tested yet, which is why none of them have been accepted upstream and released yet. But if you're not in a huge hurry, and willing to take a risk, there's a pretty good chance you can start with a fork today, and switch to the final version when it's released, hopefully before you've reached 1.0 yourself.

Second, ZeroRPC is certainly not the only RPC library available for either Python or Node. And most of them have a similar kind of interface for exposing methods over RPC. And, while you may ultimately need something like ZeroMQ for scalability or deployment reasons, you can probably use something simpler and more widespread like JSON-RPC over HTTP—which has a dozen or more Python and Node implementations—for early development, then switch to ZeroRPC later.

Third, RPC isn't exactly complicated, and binding methods to RPCs the way most libraries do isn't that hard. Making it asynchronous can be tricky, but again, for early development you can just use an easy but nonscalable solution—creating a thread for each request—and switch to something else later. (Of course that solution is only easy if your service is stateless; otherwise you're just eliminating all of your async problems and replacing them with race condition problems…)

Upvotes: 2

user3666197
user3666197

Reputation: 1

ZeroMQ offers several transport classes, while the first two will be best suited for the case of a heterogenous RPC layer

  • ipc://
  • tcp://
  • pgm://
  • epgm://
  • inproc://

ZeroMQ has ports for both systems, so will definitely serve also your projected needs.

Upvotes: 0

Related Questions