CODEWITHSUNDEEP
pythonpython-3.xpython-multiprocessing
Muhammad Ali
Muhammad Ali

Reputation: 732

python pool.map_async doesn't wait for .wait()

I'm trying to run a fat function which worked pretty well with pool.apply_async

Now, I'm trying out the pool.map_async function (with 2 arguments passed with the functools.partial method) and the program finishes immediately without any error or exceptions... 

    files = list_files(mypath) # list of files to process
    csv_rows = None
    result = mp.Queue() #result queue from multiprocessing module

    pool = mp.Pool(4)
    t = pool.map_async( partial(process_file, result), files)
    t.wait() # it doesn't wait HERE ... program exits immediately - no errors

Any clues on what I might be missing?

Upvotes: 2

Views: 5294

Answers (1)

Paul
Paul

Reputation: 5935

First off, you probably don't need map_async if you are going to immediately wait on it. If that is the case, then just use map. You can potentially also remove your queue and just return the values. But that's probably not the problem you are having.

The problem is likely that the wait method doesn't raise remote exceptions. It is likely that your process_file method is actually failing in the pool processes but that you aren't seeing those exceptions. Like Blckknght mentioned, you should switch to using a get method which, as you can see here, will raise the remote exception. Here is a simple example where the wait method hides the remote process exceptions and how if you switch to get you can see them again: 

import multiprocessing as mp

def just_dies(n):
    raise ValueError("Test")

if __name__ == "__main__":
    pool = mp.Pool(4)
    results = pool.map_async(just_dies, range(10))

    # the wait will immediately silently pass
    #results.wait()

    # this will actually raise the remote exception
    results.get()

If you run this you will get a traceback error message like so

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "foo.py", line 14, in <module>
    results.get()
  File "XXX/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
ValueError: Test

If you switch to using the wait method then you won't see it.

Upvotes: 3

Related Questions