CODEWITHSUNDEEP
pythonarraysmultiprocessingshared-memory
Vedant7
Vedant7

Reputation: 71

Multiprocessing using shared memory

Can someone provide me with sample code to share a writable array or a list among a pool of worker processes or even individually spawned processes using the multiprocessing module of python using locks? My code below spawns 2 processes of which one should print '1' and the other should print '2' to the shared array. However, when I try to print out the elements of the array after the processing it only gives me a list of 0's. Where am I going wrong? I want a writable data structure to be shared between multiple processes.

Below is my code:

import multiprocessing

arr=multiprocessing.Array('i',10,lock=True)
lock=multiprocessing.RLock()

def putitin(n):
    for i in range(5):
        lock.acquire()
        arr.append(n)
        lock.release()
    return

p1=multiprocessing.Process(target=putitin,args=(1,))
p2=multiprocessing.Process(target=putitin,args=(2,))

p1.start()
p2.start()

p1.join()
p2.join()

for i in range(10):
    print(arr[i])

Upvotes: 1

Views: 1804

Answers (2)

martineau
martineau

Reputation: 123453

One potential problem with your code is that to use multiprocessing on Windows you need to put the code for the main process in an if __name__ == '__main__': block. See the Safe importing of main module subsection of the Windows section of the multiprocessing Programming guidelines.

Another major one is that you're attempting to a share global variable among the processes. which won't work because each one runs in its own unshared memory-space, so each subprocess has its own arr. (Variables which are just module level constants are OK, though)

Lastly, a multiprocessing.Array has a fixed size and doesn't have the extend() method your code is trying to use in the putitin() function — therefore it appears you also want a writable and resizable container (that is ordered and perhaps accessible via integer indices).

In that case something the like the following may be suitable. You don't need to explicitly lock the object before making changes to it because it's a thread-safe shared object.

import multiprocessing

def putitin(lst, value):
    for i in range(5):
        lst.append(value)

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    lst = manager.list()

    p1 = multiprocessing.Process(target=putitin, args=(lst, 1))
    p2 = multiprocessing.Process(target=putitin, args=(lst, 2))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    for i in range(len(lst)):
        print(lst[i])

Output:

1
1
1
1
1
2
2
2
2
2

Upvotes: 2

Stanislau Hlebik
Stanislau Hlebik

Reputation: 11

A few issues that I found in your code. First of all, it seems to be a good idea to pass all shared resources to child and use if __name__ == '__main__'. Secondly, I don't think multiprocessing.Array has append() method (at least it didn't work for me on Python2 and Python3). Thirdly, since you are using lock=True I don't think you need additional locks.

If you need to append values to the Array I'd use separate Counter variable (see the code below).

import multiprocessing


def putitin(n, arr, counter):
        for i in range(5):
            with counter.get_lock():
                index = counter.value
                counter.value += 1
            arr[index] = n

if __name__ == '__main__':
        arr = multiprocessing.Array('i', 10,lock=True)
        counter = multiprocessing.Value('i', 0, lock=True)
        p1 = multiprocessing.Process(target=putitin,args=(1, arr, counter))
        p2 = multiprocessing.Process(target=putitin,args=(2, arr, counter))

        p1.start()
        p2.start()

        p2.join()
        p1.join()

        for i in range(10):
            print(arr[i])

Having said that, what is your high-level goal? What are you trying to achieve with it? Maybe it's possible to use multiprocessing.Pool?

Upvotes: 0

Related Questions