CODEWITHSUNDEEP
python-3.xlinuxwindowsmultiprocessing
HaroldDeng
HaroldDeng

Reputation: 11

Python3, behavior of multiprocessing module under different enviorments

I have the following code and try to run on both Linux and Windows. On Linux, the code runs good but gave me a runtime error on Windows. I also try to run the code on the online compilers, the code runs perfectly fine.

The problem is somehow related to the self keyword.

I don't own any Apple product, and it will be helpful if someone can run it on Mac OS.

Many thanks in advance.

The environment:

 class A:
    def __init__(self):

      # ==== case 1 =====
      # Ubuntu: ok, Windows: ok
      a = multiprocessing.Process(target=self.t, args=())
      a.start()
      a = multiprocessing.Process(target=self.t, args=())
      a.start()
      # ================

      # ==== case 2 ====
      # Ubuntu: ok, Windows: ok
      self.b = multiprocessing.Process(target=self.t, args=())
      self.b.start()
      self.b = multiprocessing.Process(target=self.t, args=())
      self.b.start()
      # ================

      # ==== case 3 ====
      # Ubuntu: ok, Windows: Runtime Error
      c = multiprocessing.Process(target=self.t, args=())
      c.start()
      self.d = multiprocessing.Process(target=self.t, args=())
      self.d.start()
      # ================

      # ==== case 4 ====
      # Ubuntu: ok, Windows: ok
      self.e = multiprocessing.Process(target=self.t, args=())
      self.e.start()
      f = multiprocessing.Process(target=self.t, args=())
      f.start()
      # ================

      # ==== case 5 ====
      # Ubuntu: ok, Windows: Runtime Error
      self.g = [multiprocessing.Process(target=self.t, args=()) for _ in range(2)]
      for proc in self.g:
          proc.start()
      # ================

      # ==== case 6 ====
      # Ubuntu: ok, Windows: ok
      h = [multiprocessing.Process(target=self.t, args=()) for _ in range(2)]
      for proc in h:
          proc.start()
      # ================

      time.sleep(1) 

    def t(self):
        print("Hi")


if __name__ == "__main__":
    A()

The error that is given by Python on Windows when running case 5 alone.

Traceback (most recent call last):
  File ".\GeneralUse.py", line 52, in <module>
    A()
  File ".\GeneralUse.py", line 42, in __init__
    proc.start()
  File "C:\Program Files\Python3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python3\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python3\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python3\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Upvotes: 1

Views: 828

Answers (1)

miksus
miksus

Reputation: 3417

This is not OS related problem. The problem is the Python version. I'm running your code on Windows with Python 3.6.6 and it works fine but fails for Python 3.9 (for Windows). You also have 3.6 for Ubuntu but 3.8 for Windows thus indicating for this.

The issue arises when you start another Process and feed the previously started process to this. This is not a problem for passing not started processes. This is done via self in your case: self is pickled during the starting of the process.

I answered the same issue here: https://stackoverflow.com/a/65749012/13696660

In short, I think the cleanest solution is to remove the problematic processes from pickling the object:

class A:
    def __init__(self):
        ...

    def __getstate__(self):
        # capture what is normally pickled
        state = self.__dict__.copy()

        # remove unpicklable/problematic variables
        # (multiprocessing.Process in this case)
        state['b'] = None
        state['d'] = None
        state['e'] = None
        state['g'] = None
        return state
...

Now it works on my Windows for 3.9.

Upvotes: 1

Related Questions