CODEWITHSUNDEEP
pythonrpc
redShadow
redShadow

Reputation: 6777

Python interprocess communication

I have a Python application (with GUI, using PyQt4) that gets spawned by the user in several instances. The application is used to execute some long running tasks (about some hours to a few days), so I'm willing to add an extra "monitoring" application that will do things like:

One way that would fit the job is RPyC, the only problem, it seems to work only over TCP sockets, like most of the RPC libraries I found. But this leads to having to open several unneeded sockets listening only on localhost, and having to create some kind of ports allocation mechanism to avoid two processes trying to listen on the same port. And then the monitor needs a list of ports to be written somewhere, or to go and find processes listening on TCP ports and try to figure out whether they are instances of the correct application or ot. Sounds like a mess.

The nicest way of managing the intercommunication I could think of at the moment would be having some unix sockets, let's say, in '/var/run/myapp/myapp-.sock', and then creating a module that does all the dirty stuff exposing some methods like listMyApps() and getMyApp(pid) returning the first a list of pids, the second an object that could be used for communication with that app.

Now, I'm looking at the best way to accomplish that.. For real, there isn't anything out there already done to manage RPC over unix sockets? Sounds a bit strange to me, but I wasn't able to find anything that could fit..

Any suggestions?

NOTE: I don't want to reverse things (applications are client of a single monitoring server) to avoid problems in case of monitoring application crash, and to letting me free to create other applications that connect to these sockets and make requests.

NOTE: Security is not an issue since all this stuff is running in a private, closed and firewalled network :), plus, requests are done on localhost by trusted users only.

Upvotes: 3

Views: 3744

Answers (3)

redShadow
redShadow

Reputation: 6777

I guess I found a soultion by using dbus-python. I was able to integrate it with the Qt4 mainloop, although on the website they say only glib was supported (I guess that page is not updated). I just did a qick test and it seems to be working nicely (I just esposed some dummy functions on a bus named com.example.myapp.<pid> and listed instances and connected from an external client).

Then, I could use the RPyC stuff over TCP only for the communication between a "master" control application and the "manager" application that acts quite like a "switch" between instances on a machine.

Some ascii-art clarification:

+-------------------+                    
| MASTER APP        |                    +--------------------+
| on my workstation |------ RPyC --------| Server#0 Manager   |
+-------------------+                    +--------------------+
      |                                      | | |
    RPyC                                     | | '-- dbus ---[INSTANCE #0]
      |                                      | '--- dbus ---[INSTANCE #1]
   +--------------------+                    '---- dbus ---[INSTANCE #2]
   | Server#1 Manager   |
   +--------------------+
      | | |
      | | '-- dbus ---[INSTANCE #0]
      | '--- dbus ---[INSTANCE #1]
      '---- dbus ---[INSTANCE #2]

If anyone is interested, just let me know and I'll post some code examples/more details too..

Upvotes: 2

btilly
btilly

Reputation: 46497

Another way to go is to have some sort of job control system, and instead of having the application start its own processes, have it register jobs to be started that do the task. Then the job control system would be monitoring the jobs that are started. As a bonus, with this design you can spread work out across many machines at some point in the future.

Upvotes: 2

Falmarri
Falmarri

Reputation: 48587

But this leads to having to open several unneeded sockets listening only on localhost, and having to create some kind of ports allocation mechanism to avoid two processes trying to listen on the same port

Not quite. First, that's how most interprocess communication works, through sockets. Either TCP sockets or UNIX sockets. That's essentially (not exactly) what you're doing when piping standard out and such.

You could also use OS signals. Although you have to keep in mind that only the main thread of each process can do signal handling, so you have to be careful to not block that.

But any way you do it there's pretty much no getting around using socket connections.

Upvotes: 2

Related Questions