CODEWITHSUNDEEP
pythonmultiprocessing
user14175128
user14175128

Reputation: 29

How to terminate multiprocessing (under Linux) after an exception has raised in subordinate task?

The following code shows a sample program that raises an exception in a multiprocessing task. If I run it (in an Linux environment), the script / program never terminates.

What do I have to do to terminate the program successfully / proceed further after the exception occurred in the task?

from multiprocessing import Pool
from multiprocessing import cpu_count
import traceback
import sys

def myTask(runNumber, _):
    if runNumber > 6:
        raise ValueError(runNumber)
    else:
        return runNumber

def on_error(e):
    traceback.print_exception(type(e), e, e.__traceback__)
    global terminated
    terminated = True
    pool.terminate()
    print(f"oops:{e}")
    sys.exit()

if __name__ == '__main__':
    myCpuCount = cpu_count()
    print("myCpuCount=", myCpuCount)
    pool = Pool(myCpuCount)
    myResult = [pool.apply_async(myTask, (runNumber, 1), error_callback=on_error)
                for runNumber in range(15)]

    pool.close()
    pool.join()

    myResult = [r.get() for r in myResult]

    print("myResult=", myResult)

The output:

/home/user/venv/numba/bin/python3.6 /home/user/PycharmProjects/myProject/attic03.py
myCpuCount= 4
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/user/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/user/PycharmProjects/myProject/attic03.py", line 10, in myTask
    raise ValueError(runNumber)
ValueError: 7
"""

The above exception was the direct cause of the following exception:

ValueError: 7
oops:7

I can see that the program is still executed and does not end at all.

Upvotes: 0

Views: 63

Answers (1)

Giannis
Giannis

Reputation: 5526

Not sure what the issue is without debugging, but I'd start with removing sys.exit and pool.teeminate calls from the on_error. These may be interfering with the pool.join. Also should not need the pool.close. Related question: how to to terminate process using python's multiprocessing

Upvotes: 1

Related Questions