CODEWITHSUNDEEP
pythonpython-3.xprocessoperating-systempython-3.8
Virgil_S
Virgil_S

Reputation: 43

Multiprocessing in python3 get different values when running on mac and linux

Multiprocessing in Python 3 get different values when running on Mac and Linux.

For example, when forking the variable values in the main process should be cloned to the child process. But after testing, Linux's result is different from Mac.

Here is the code 

#!/usr/bin/env python
# coding=utf-8
import multiprocessing

test_temp = "a"


def check_value_in_other_process():
    print(f"[{multiprocessing.current_process().name}] test_temp={test_temp}")


def change_value():
    global test_temp
    test_temp = "b"


if __name__ == '__main__':

    print(f"[{multiprocessing.current_process().name}] origin test_temp={test_temp}")
    change_value()
    print(f"[{multiprocessing.current_process().name}] changed test_temp={test_temp}")

    pool = multiprocessing.Pool(4)
    pool.apply_async(func=check_value_in_other_process)
    pool.close()
    pool.join()

On Linux，I tested with Debian 9 and CentOS 8, the result is 

[MainProcess] origin test_temp=a
[MainProcess] changed test_temp=b
[ForkPoolWorker-1] test_temp=b

On mac, I tested with mac os 14 and mac os 15, the result is：

[MainProcess] origin test_temp=a
[MainProcess] changed test_temp=b
[SpawnPoolWorker-2] test_temp=a

Maybe,the difference is caused by ForkPoolWorker and SpawnPoolWorker?

After check,I found that there is 3 ways to create process:spawn、fork、forkserver. So I add:

multiprocessing.set_start_method('fork')

Linux and Mac got the same result But：

  1. Why does mac act the different default behavior?
  2. How can I set the default behavior in my project?

Upvotes: 4

Views: 1756

Answers (1)

Selcuk
Selcuk

Reputation: 59315

I am seeing the former output (the one you see on Linux) on macOS 10.15 with Python 3.6.5, and it turns out that they changed this behaviour in Python 3.8 (you must be running 3.8 or higher):

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

In spawn mode, the child process will only inherit those resources necessary to run the process objects run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. In fork mode (default for Unix) all resources of the parent are inherited by the child process.

To answer your second question: It is not recommended to use fork on macOS (as mentioned above), so if you must have the exact same behaviour on both platforms, stick to spawn.

Upvotes: 4

Related Questions