CODEWITHSUNDEEP
pythonsynchronizationmultiprocessing
Aphrodite
Aphrodite

Reputation: 121

How to sync a variable with multiple processes in python?

I wouldd like to use multiprocessing pool to access, read, and modify a variable in python. The processes are completely separate, but have to access the same variable. This would be easily done with pointers in C, but how do I do it in python?

For example, I have done this:

import multiprocessing as mp
import time
a=3

def one(arg):
        time.sleep(2*arg[1])
        print(arg)
        arg[0]+=1
        print(arg[0])
        return arg[0]

if __name__ == '__main__':
        with mp.Pool(processes=2) as pool:
                print(pool.map(one,[[a,1],[a,2]]))

But it returns 

[3, 1]

4

[3, 2]

4

[4, 4]

Instead of [4,5].

Upvotes: 3

Views: 3227

Answers (1)

CodeSamurai777
CodeSamurai777

Reputation: 3355

You can share variables between processes. Using shared memory.

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

Value and Array are special objects that allows you to share state between them. The d and i are typcodes of variable type you want to share.

However, this solution doesn't provide any sync mechanism. You need to be careful not to read and write at the same time.

Also this solution can be used only for simple values. If you need to share object you should look at Manager class in multiprocessing module.

An exert from docs

A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array

So as you see you can also share RLock to set up a sync mechanism for reading and writing your shared variables.

More info here: https://docs.python.org/3.5/library/multiprocessing.html#sharing-state-between-processes

Upvotes: 2

Related Questions