How to make multiple python programs communicate in this situation?

Question

I'm a bit new at Python and I am working on a robotics project. The short form of my question is that I am trying to find the best way (for my situation) to run multiple python programs at once.

A little bit of context, my robot is a platform for a service robot that is capable of following markers and paths using image algorithms and also receive commands from a remote computer. I want to have separate programs for the image processing, the driving, and so on, and then manage all of them through a main program. I know I can't use anything basic like functions or classes, because each of these processes must be looping continuously, and I don't want to combine all the code to run in a single while loop, because it runs very slowly and it is significantly harder to manage.

So, in short, how do I make two separate, looping programs "talk"? Like I want the imaging program to send information about what it sees to the driving and steering program, etc.

I did some research and I found some information on multithreading and API's and stuff like that, though I can't really tell which one is actually the thing I'm looking for.

To clarify, I just need to be pointed in the right direction. This doesn't seem like a very high-level thing, and I know there are definitely tutorials out there, I'm just really confused as to where to start as I am teaching myself this as I go.

Answer 1

This is probably a little bit outside the scope of your project, but have you considered using ROS? It lets you run a bunch of different nodes (can be Python scripts) at the same time that communicate by publishing and subscribing to topics. They can be on the same system (i.e. one or more nodes on the robot) or different systems (i.e. one node on the robot, multiple nodes on the PC). ROS also has a lot of awesome built in tools and libraries that are specifically made for robotic systems such as visualization, mapping, odometry, etc. Here's a bit of starting info:

https://en.wikipedia.org/wiki/Robot_Operating_System

http://wiki.ros.org/ROS/StartGuide

It's usually used for much larger frameworks than you seem to be describing, and beware that it takes quite a bit of time (in my experience) to implement, but it is very easy to expand once its up and running. Like I said, it all depends on the scope of your project!

Good luck!

Answer 2

After some sniffing around, I found that using IPC was a good solution. The process I used wasn't too difficult, I just made some very simple server and client classes and had them communicate over the Localhost IP. There's undoubtedly a better way to do this, but for a beginner like myself, it was a simple way to make two programs talk without modifying code too much. For those who are trying to do a similar thing as I did, here's the classes I made for myself. Fair warning, they're not exactly pristine or even very complex, but they got the job done.

Here's the class I made for the server:

import socket
from random import random
from time import sleep

class ServerObject:

    def __init__(self,host_address,port):

        self._host_address = host_address
        self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self._s.bind((self._host_address,port))

    def handshake(self):

        print "Server Started. Awaiting Connection"

        while True: 
            _data, _addr = self._s.recvfrom(1024)
            if str(self._s.recvfrom(1024)[0]) == 'marco':
                break

        print 'marco recieved. sending polo...'

        while True:
            self._s.sendto('polo',_addr)
            if str(self._s.recvfrom(1024)[0]) == 'confirm':
                break
            sleep(.5)

        print 'connection verified'

        self._addr = _addr

        return True

    def send(self,data):

        self._s.sendto(str(data),self._addr)

    def recieve(self,mode = 0):

        _data, _addr = self._s.recvfrom(1024)

        if mode == 0:
            return str(_data)
        if mode == 1:
            return int(_data)
        if mode == 2:
            return float(_data)
        if mode == 3:
            return tuple(_data)

    def change_port(self,port):

        self._s.bind((self._host_address,port))

    def close(self):
        self._s.close()
        print '_socket closed_'


if __name__ == '__main__':

    host = '127.0.0.1'

    talk = ServerObject(host,6003)
    talk.handshake()

And here's the class I made for the client:

import socket
from time import sleep

class ClientObject:

    def __init__(self,host_address,server_port,port = 0):

        self._server = (host_address,server_port)
        self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self._s.bind((host_address,port))


    def handshake(self):

        print ' sending marco'

        self._s.sendto('marco',self._server)
        sleep(.1)
        self._s.sendto('marco',self._server)

        while True:
            if str(self._s.recvfrom(1024)[0]) == 'polo':
                break

        #self._s.sendto('marco',self._server)
        #self._s.sendto('marco',self._server)

        print ' connection verified'
        self._s.sendto('confirm',self._server)

        self._s.setblocking(0)

        return True

    def recieve(self,mode = 0):

        _data, _addr = self._s.recvfrom(1024)

        if mode == 0:
            return str(_data)
        if mode == 1:
            return int(_data)
        if mode == 2:
            return float(_data)
        if mode == 3:
            return tuple(_data)

    def send(self,data):
        self._s.sendto(str(data),self._server)

    def close(self):
        self._s.close()
        print '_socket closed_'


if __name__ == '__main__':

    host = '127.0.0.1'
    port = 0

    talk = ClientObject(host,24603,port)
    talk.handshake()

    #while True:
        #print talk.recieve()

Use the ServerObject class on the program that will primarily send data and the ClientObject class on the program that will primarily recieve data. These can be flipped around in many situations, but I found it's best to do it this way to take advantage of UDP. The client class has an optional port variable that is set to 0 by default. This is because for UDP the client needs another port to establish itself on. 0 means it will pick an available port, but if you specify one, it's possible to re-establish a connection if the client goes offline without needing to restart both programs.

Use the handshake first on both programs being sure to use the same IP and port (not referring to the last variable on the client) and then use the send and receive functions to pass data back and forth.

again, these aren't that good, in fact there's many problems that cab arise with using this method, but for a simple task, they got the job done. I set up the handshake to print verifications of what is happening, but if those get annoying, you can just remove those lines.

Hope this helps!

Answer 3

I think multiproccessing library could be a solution. You will be able to run several processes in parallel when each process could perform it specific work, while sending data to each other.

You can check this example

This is generic directory walker, which have process that scans directory tree and passes the data to other process, which scans files in already discovered folders. All this done in parallel.