CODEWITHSUNDEEP
pythonlinuxmultiprocessing
Richard
Richard

Reputation: 61559

Kill Python Multiprocessing Pool

I am running a Python program which uses the multiprocessing module to spawn some worker threads. Using Pool.map these digest a list of files.

At some point, I would like to stop everything and have the script die.

Normally Ctrl+C from the command line accomplishes this. But, in this instance, I think that just interrupts one of the workers and that a new worker is spawned.

So, I end up running ps aux | grep -i python and using kill -9 on the process ids in question.

Is there a better way to have the interrupt signal bring everything to a grinding halt?

Upvotes: 23

Views: 12061

Answers (4)

phihag
phihag

Reputation: 288298

SIGQUIT (Ctrl + \) will kill all processes even under Python 2.x.

You can also update to Python 3.x, where this behavior (only child gets the signal) seems to have been fixed.

Upvotes: 26

dano
dano

Reputation: 94981

Unfortunately in Python 2.x there really isn't a good solution for this problem. The best workaround that I know of is to use pool.map_async(...).get(timeout=<large number>) instead of pool.map. The problem is that pool.map makes a call to threading.Condition.wait(), which for some reason can't be interrupted by Ctrl+C in Python 2.x (it works in Python 3). When you use map_async(), it calls threading.Condition.wait(timeout=<large number>), which ends up doing a busy wait loop, which can be interrupted by Ctrl+C.

Try it for yourself:

c = threading.Condition()
try:
    c.acquire()
    c.wait()  # You won't be able to interrupt this
except KeyboardInterrupt:
    print("Caught it")

c = threading.Condition()
try:
    c.acquire()
    c.wait(timeout=100)  # You CAN interrupt this
except KeyboardInterrupt:
    print("Caught it")

So, to make your map call interruptable, do this:

if __name__ == "__main__":
    p = multiprocessing.Pool()
    try:
        p.map_async(func, iterable).get(timeout=10000000)
    except KeyboardInterrupt:
        print("Caught it")
        # Optionally try to gracefully shut down the worker processes here.
        p.close()
        # DON'T join the pool. You'll end up hanging.

Also note, as pointed out by phihag, this issue is fixed in Python 3.4 (and possibly earlier in 3.x).

Upvotes: 4

SimplyMagisterial
SimplyMagisterial

Reputation: 36

I found that using the python signal library works pretty well in this case. When you initialize the pool, you can pass a signal handler to each thread to set a default behavior when the main thread gets a keyboard interrupt.

If you really just want everything to die, catch the keyboard interrupt exception in the main thread, and call pool.terminate().

Upvotes: 0

ollien
ollien

Reputation: 4796

There are a couple of ways. The first way is to mark the thread as a daemon using

in Threading,

myThread.setDaemon(true)

in multiprocessing,

myThread.daemon = True

All threads marked as a daemon will terminate with the main thread. This is not the proper way to do it as it doesn't allow the threads to clean up

The next way is to listen for KeyboardInterrupt with a try-catch, and then .join() the threads like such.

try:
    myThread = MyThread()
except KeyboardInterrupt:
    myThread.join()

If your thread is in a loop, you could use a condition such as a boolean, set this to false, and when the condition is false, it performs cleanup.

class MyThread(Threading.thread):
    def __init__(self):
        self.alive=True
    def run(self):
        while self.alive:
            #do stuff
        #cleanup goes here, outside the loop
try:
    myThread = MyThread()
except KeyboardInterrupt:
    myThread.alive = False
    myThread.join()

Upvotes: 3

Related Questions