CODEWITHSUNDEEP
pythonmultiprocessing
Kyle Sponable
Kyle Sponable

Reputation: 735

Good way to control python processes

Where is a good place to see best practice for controlling processes.

Specifically I want to know how to give a shutdown command. I was thinking a global boolean that each process polls

Also a good way to pass data to and from and in-between active processes.

I have not found any good blog posts they are mostly how-tos

Upvotes: 1

Views: 149

Answers (1)

freakish
freakish

Reputation: 56557

The idea of a global boolean is quite good, except that you can't share it between processes. But you can keep a copy of that boolean in each subprocess.

One way would be to use pipes and/or queues together with threads. Something like that:

from multiprocessing import Process, Pipe
from threading import Thread

Alive = True    

def listener_thread(conn):
    global Alive
    while True:
        data = conn.recv()
        if data == "kill":
            Alive = False
            break

def subprocess(conn):
    t = Thread(target=listener_thread, args=(conn,))
    t.start()
    while Alive:
        # do some stuff here

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=subprocess, args=(child_conn,))
    p.start()
    # do some stuff
    parent_conn.send("kill")
    p.join()

Note that you will need a (parent_conn, child_conn) pair per each subprocess. That is a standard way of communicating between process and subprocess.

Or you could use a database (and by database I mean any database, even a simple file) to share data between processes. However you will need a polling thread for that which might not be efficient.

You could also use some pub/sub system (so you won't have to worry about polling efficiency) like Redis which could be the best depending on your needs.

The less processes share the better in general.

Upvotes: 2

Related Questions