CODEWITHSUNDEEP
pythonglobal-variablespython-multiprocessing
George Pligoropoulos
George Pligoropoulos

Reputation: 2939

Are global variables get replicated in each process when doing multiprocessing in Python?

We have used parallel processing by having some functions being called by runInParallel that you will find in this answer: https://stackoverflow.com/a/7207336/720484

All of these functions are supposed to have access to a single global variable which they should read.
This global variable is actually an instance of a class. This instance contains a member variable/attribute and all of the processes read and write to it.

However things are not happening like this. The object(class instance) seems to be replicated and that its attributes are independent on each process. So if one process changes the value this is not visible to the variable of the other process.

Is this the expected behavior?

How to overcome it?

Thank you

Upvotes: 1

Views: 809

Answers (1)

Sergey Vasilyev
Sergey Vasilyev

Reputation: 4189

All children processes will inherit that instance at the moment of forking from the parent process. Any changes made to the instance in the children and in the parent will NOT be seen after the fork.

This is how the processes work in Linux — every process has its own memory, protected from other processes (unless you intentionally shared it). It is not Python-specific.

What you are looking for is called IPC (Inter-Process Communication) in general. There are multiple ways how the processes can communicate with each another. You might want to use pipes or the shared memory.

In Python, read this: https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes

Upvotes: 4

Related Questions