CODEWITHSUNDEEP
pythonmultithreadingpython-3.xpython-multiprocessingpython-multithreading
mxscho
mxscho

Reputation: 2170

Python 3: Termination of threads in multiprocessing ThreadPool not working

I am using Ubuntu 16.04.2 LTS and Python 3.5.2. I have a ThreadPool from multiprocessing doing some work. Now I want to terminate this pool while doing its work.

Using ThreadPool.terminate() does not work as expected. When running the following example, the worker never stops doing its work and the program never runs past the ThreadPool.join() call.

import time
from multiprocessing.pool import ThreadPool

def task():
    try:
        while True:
            print("Working")
            time.sleep(1)
    except: # Unsuccessful attempt. :(
        print("Working, stopping now")

thread_pool = ThreadPool(processes=1)

thread_pool.apply_async(task)
time.sleep(1) # Make sure the task is actually started.

print("Terminating")
thread_pool.terminate()
print("Termination: Initiated")
thread_pool.join() # Does not return.
print("Termination: Done")

What am I doing wrong?

Upvotes: 0

Views: 3365

Answers (1)

torek
torek

Reputation: 488163

You specifically requested a ThreadPool rather than a Pool. This means that instead of creating a new process, the multiprocessing code will create a local thread (in your own process).

A ThreadPool instance cannot be terminated abruptly as there is no defined mechanism for that (vs a process, where the system can call os.kill). In your code, the .terminate directive winds up being ignored. In fact, it sets a flag that the pool instance would check when the task returns, but your task never returns. We can see this with a revised version:

import argparse
import sys
import time

from multiprocessing.pool import Pool, ThreadPool

def task(arg):
    for i in range(3):
        print("Working on", arg, "with i =", i)
        time.sleep(1)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--delay', default=1, type=float)
    args = parser.parse_args()

    thread_pool = ThreadPool(processes=1)

    thread_pool.apply_async(task, (0,))
    thread_pool.apply_async(task, (1,))
    time.sleep(args.delay)

    print("Terminating")
    thread_pool.terminate()
    print("Termination: Initiated")
    thread_pool.join() # Does not return.
    print("Termination: Done")

if __name__ == '__main__':
    try:
        sys.exit(main())
    except KeyboardInterrupt:
        sys.exit('\nInterrupted')

which does this when run with no arguments:

$ python3 tp.py
Working on 0 with i = 0
Working on 0 with i = 1
Terminating
Termination: Initiated
Working on 0 with i = 2
Termination: Done

and this when run with --delay 5:

$ python3 tp.py --delay 5
Working on 0 with i = 0
Working on 0 with i = 1
Working on 0 with i = 2
Working on 1 with i = 0
Working on 1 with i = 1
Terminating
Working on 1 with i = 2
Termination: Initiated
Termination: Done

Alternatively, if you just use a Pool instead of a ThreadPool, you will get a real process and can use .terminate on it at any time (modulo the usual caveats about screwing up communications queues).

Upvotes: 1

Related Questions