CODEWITHSUNDEEP
pythonpython-3.x
Pep_8_Guardiola
Pep_8_Guardiola

Reputation: 5252

All example concurrent.futures code is failing with "BrokenProcessPool"

I am trying to get a basic understanding of this before I create the actual application I need. I recently moved over from 2.7 to 3.3.

A direct copy-paste of this code from the python docs fails, as does a slightly simpler example from here.

This is my code, derived from the second example:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x

# Make sure the map and function are working
print([val for val in map(f, nums)])

# Test to make sure concurrent map is working
with concurrent.futures.ProcessPoolExecutor() as executor:
    for item in executor.map(f, nums):
        print(item)

And this is the output:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Traceback (most recent call last):
  File "<string>", line 420, in run_nodebug
  File "<module1>", line 13, in <module>
  File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator
    yield future.result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

How can I get this code to work as expected? I was hoping the examples would just work out of the box.

Upvotes: 60

Views: 96368

Answers (3)

ebonnal
ebonnal

Reputation: 1167

Note: Using the pool directly from the body of a script will be problematic for all systems (even fork-based ones) as of python 3.14:

Note The default multiprocessing start method (see Contexts and start methods) will change away from fork in Python 3.14. Code that requires fork be used for their ProcessPoolExecutor should explicitly specify that by passing a mp_context=multiprocessing.get_context("fork") parameter.

https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

Upvotes: 0

rakesh
rakesh

Reputation: 4498

Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using processpoolexecutor or any other parallel code which spawns new processes.

Basically, all your code which creates new processes must be under if __name__ == '__main__': , for the same reason you cannot execute it in interpreter.

Upvotes: 44

Pep_8_Guardiola
Pep_8_Guardiola

Reputation: 5252

This was my fault, for two reasons:

  1. The code was un-guarded, i.e no if __name__
  2. The strange looking Traceback was because the file was not saved. Never caused me an issue before, but did in this case.

Correcting both of those fixed the error.

Final test code:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x
def main():
    # Make sure the map and function are working
    print([val for val in map(f, nums)])

    # Test to make sure concurrent map is working
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print([val for val in executor.map(f, nums)])

if __name__ == '__main__':
    main()

Output, as expected:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Upvotes: 82

Related Questions