CODEWITHSUNDEEP
pythondictionarymultiprocessing
nam
nam

Reputation: 3632

share dict between process

I have a dict that I want to share with 4 processes that modify values in that dict in real time. The dict is created with Manager().dict() The question is, do I need to lock the resource every time a process modify the dict? What if 4 processes access to this dict in real time?

Upvotes: 1

Views: 1458

Answers (1)

Tim Peters
Tim Peters

Reputation: 70602

You'll see undefined behavior then. Here's a simple test program:

def worker(t):
    d, i = t
    d[i % 10] += 1

if __name__ == "__main__":
    import multiprocessing as mp
    pool = mp.Pool()
    d = mp.Manager().dict()
    for i in range(10):
        d[i] = 0
    pool.map(worker, ((d, i) for i in xrange(1000)))
    pool.close()
    pool.join()
    print d, sum(d.values())

And here's sample output from 3 runs:

{0: 97, 1: 96, 2: 98, 3: 96, 4: 96, 5: 99, 6: 97, 7: 96, 8: 96, 9: 94} 965
{0: 97, 1: 97, 2: 96, 3: 97, 4: 97, 5: 97, 6: 95, 7: 95, 8: 93, 9: 96} 960
{0: 98, 1: 97, 2: 98, 3: 96, 4: 97, 5: 95, 6: 97, 7: 97, 8: 97, 9: 98} 970

To get the "expected" count of 100 in each bucket, you need to create a mp.Manager().Lock() object too, pass it, and use it in the worker() to protect the dict mutation.

Upvotes: 3

Related Questions