CODEWITHSUNDEEP
pythonparallel-processingmultiprocessingfuture
Evator
Evator

Reputation: 26

ProcessPoolExecutor gets stuck with error: A process in the process pool was terminated abruptly while the future was running or pending

I tried to make a basic program with multiprocessing in Python but I can't get it to work with the concurrent.futures library. Other posts I have found on the topic did not help me either. The program just gets stuck after starting the futures. I have no idea what the reason for that could be. This is my code:

import concurrent.futures

def futureFunction():

    print("inside the future")

    return 0


def myMethod():

    print("start pool...")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        futs = []

        for i in range(4):
            print("start future", i)
            futs.append(executor.submit(futureFunction))

        for f in concurrent.futures.as_completed(futs):

            print("done", f.result())

myMethod()

And this is the output I get:

start pool...
start future 0
start future 1
start future 2
start future 3

Why does this happen? Thanks!

EDIT: I get this Error when running from a .py file (I am using python 3.6)

start pool...
start future 0
start future 1
start future 2
start future 3
start pool...
start future 0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 18, in myMethod
start pool...
    futs.append(executor.submit(futureFunction))
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 466, in submit
start pool...
    self._start_queue_management_thread()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 427, in _start_queue_management_thread
    self._adjust_process_count()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 446, in _adjust_process_count
start future 0
start future 0
    p.start()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
Traceback (most recent call last):
  File "<string>", line 1, in <module>
start pool...
    _check_not_importing_main()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
start pool...
start future 0
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
start future 0
start future 0
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    prepare(preparation_data)
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
start pool...
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    _fixup_main_from_path(data['init_main_from_path'])
  File "<string>", line 1, in <module>
    _fixup_main_from_path(data['init_main_from_path'])
  File "<string>", line 1, in <module>
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
start future 0
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 295, in _queue_management_worker
    shutdown_worker()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 253, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 129, in put_nowait
    return self.put(obj, False)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 83, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 22, in myMethod
    print("done", f.result())
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

C:\Users\pcuser\Desktop>test.py
start pool...
start future 0
start future 1
start future 2
start future 3
start pool...
start future 0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 18, in myMethod
    futs.append(executor.submit(futureFunction))
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 466, in submit
    self._start_queue_management_thread()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 427, in _start_queue_management_thread
    self._adjust_process_count()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 446, in _adjust_process_count
    p.start()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
start pool...
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
start pool...
start future 0
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
start future 0
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    exitcode = _main(fd)
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    _fixup_main_from_path(data['init_main_from_path'])
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
start pool...
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
start future 0
    pkg_name=pkg_name, script_name=fname)
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
start future 0
Traceback (most recent call last):
    run_name="__mp_main__")
    mod_name, mod_spec, pkg_name, script_name)
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    exec(code, run_globals)
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 295, in _queue_management_worker
    shutdown_worker()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 253, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 129, in put_nowait
    return self.put(obj, False)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 83, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 22, in myMethod
    print("done", f.result())
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

C:\Users\pcuser\Desktop>test.py
start pool...
start future 0
start future 1
start future 2
start future 3
start pool...
start future 0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
start future 0
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
Traceback (most recent call last):
start pool...
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    pkg_name=pkg_name, script_name=fname)
start pool...
start future 0
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 18, in myMethod
Traceback (most recent call last):
    futs.append(executor.submit(futureFunction))
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 466, in submit
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
start future 0
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
    self._start_queue_management_thread()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 427, in _start_queue_management_thread
    self._adjust_process_count()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 446, in _adjust_process_count
  File "<string>", line 1, in <module>
Traceback (most recent call last):
    pkg_name=pkg_name, script_name=fname)
    p.start()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    self._popen = self._Popen(self)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    mod_name, mod_spec, pkg_name, script_name)
    return _default_context.get_context().Process._Popen(process_obj)
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
start pool...
    prepare(preparation_data)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    exec(code, run_globals)
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 18, in myMethod
    return Popen(process_obj)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    _fixup_main_from_path(data['init_main_from_path'])
start future 0
    futs.append(executor.submit(futureFunction))
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    prep_data = spawn.get_preparation_data(process_obj._name)
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 466, in submit
Traceback (most recent call last):
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
start pool...
    run_name="__mp_main__")
    self._start_queue_management_thread()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 427, in _start_queue_management_thread
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
start future 0
    _check_not_importing_main()
start pool...
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
  File "<string>", line 1, in <module>
    self._adjust_process_count()
start future 0
    run_name="__mp_main__")
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
Traceback (most recent call last):
start future 0
    pkg_name=pkg_name, script_name=fname)
start future 0
start future 0
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 446, in _adjust_process_count
start pool...
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
Traceback (most recent call last):
    is not going to be frozen to produce an executable.''')
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
    exitcode = _main(fd)
    p.start()
start future 0
    pkg_name=pkg_name, script_name=fname)
  File "<string>", line 1, in <module>
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.start future 0
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    mod_name, mod_spec, pkg_name, script_name)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main

    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    prepare(preparation_data)
    self._popen = self._Popen(self)
    mod_name, mod_spec, pkg_name, script_name)
  File "<string>", line 1, in <module>
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    exec(code, run_globals)
  File "<string>", line 1, in <module>
    exitcode = _main(fd)
    exitcode = _main(fd)
    exitcode = _main(fd)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 295, in _queue_management_worker
    shutdown_worker()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\process.py", line 253, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 129, in put_nowait
    return self.put(obj, False)
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\multiprocessing\queues.py", line 83, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "C:\Users\pcuser\Desktop\test.py", line 24, in <module>
    myMethod()
  File "C:\Users\pcuser\Desktop\test.py", line 22, in myMethod
    print("done", f.result())
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\pcuser\AppData\Local\Programs\Python\Python36\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

EDIT 2: When I change the method call to:

if __name__ == '__main__':
    myMethod()

the problem is fixed for when I run the program from a .py file. But the problem still exists when I try to run the code inside visual studio.

Upvotes: 1

Views: 4843

Answers (1)

d4nh5u
d4nh5u

Reputation: 71

If you just want to solve this problem. You can try to use concurrent.futures.ThreadPoolExecutor(max_workers) in place of concurrent.futures.ProcessPoolExecutor().

The default setting of max_workers is based on the number of CPUs. You can check the documentation of the ThreadPoolExecutor().

If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor.

Upvotes: 7

Related Questions