CODEWITHSUNDEEP
python-3.xmultiprocessingpython-multiprocessingconcurrent.futures
hrshd
hrshd

Reputation: 1001

Runtime error using concurrent.futures.ProcessPoolExecutor

I have seen many YouTube videos for basic tutorials for concurrent.futures.ProcessPoolExecutor. I have also seen posts in SO here and here, GitHub and GitHubMemory, yet no luck.

Problem: I'm getting the following runtime error:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I admit it, I do not fully understand this error since this is my very first attempt at multiprocessing in my python code.

Here's my pseudocode:

module.py

import xyz
from multiprocessing import freeze_support

def abc():
  return x

def main():
  xyz
  qwerty

if __name__ == "__main__":
  freeze_support()
  obj = Object()
  main()

classObject.py

import abcd

class Object(object):

  def __init__(self):
    asdf
    cvbn
    with concurrent.futures.ProcessPoolExecutor(max_workers=2) as  executor:
      executor.map(self.function_for_multiprocess, var1, var2)
      # ****The error points at the code above.👆*👆*👆 

  def function_for_multiprocess(var1, var2):
    doSomething1
    doSomething2

    self.variable = something

My class file (classObject.py) does not have the "main" guard.

Things I have tried:

  1. Tried adding if __name__ == "__main__": and freeze_support in the classObject.py along with renaming __init__() to main()`
  2. While doing the above, removed the freeze_support from the module.py

I haven't found a different solution from the link provided above. Any insights would be greatly appreciated!

I'm using a MacBook Pro (16-inch, 2019), Processor 2.3 GHz 8-Core Intel Core i9, OS:Big Sur. I don't think that matters but just declaring it if it does.

Upvotes: 2

Views: 2067

Answers (1)

Dariyoush
Dariyoush

Reputation: 508

you need to pass arguments as picklable object, so as list or a tuple. and you don't need freeze_support() just change executor.map(self.function_for_multiprocess, var1, var2) to executor.map(self.function_for_multiprocess, (var1, var2))

from multiprocessing import freeze_support
import concurrent.futures

class Object(object):

    def __init__(self, var1=1, var2=2):
        with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
            executor.map(self.function_for_multiprocess, (var1, var2))

    def function_for_multiprocess(var1, var2):
        print('var1:', var1)
        print('var2:', var2)

def abc(x):
    return x

def main():
    print('abc:', abc(200))

if __name__ == "__main__":
    #freeze_support()
    obj = Object()
    main()

Upvotes: 1

Related Questions