CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
Sergey Fedorov
Sergey Fedorov

Reputation: 2169

How to share dictionary between threads using multiprocessing.Pool?

I'm trying to share a dictionary between threads of multiprocessing.Pool. However, I'm failed with current implementation. Test code:

#!/usr/bin/env python

import multiprocessing

success_map = {"1": []}

def run_(input):
    global success_map
    successes = success_map.get(input);
    successes.append(0)
    print success_map

pool = multiprocessing.Pool()
pool.map(run_, ["1"])
pool.close()
pool.join()
print success_map

The output is 

{'1': [0]}
{'1': []}

It looks to me that worker(s) of multiprocessing.Pool() create copies of dictionary; that's why I can't see update when processing is finished. Am I right?

Note:

  1. I know it's possible to return value from run_ function and get collected list as a result of pool.map(run_, ["1"]), but need to use global variable for current task
  2. I'm aware of possible dataraces
  3. I've read somewhere that global statement isn't needed in that case but example works the same way without it
  4. I've got the same result passing [["1", success_map]] to pool.map; without usage of global variable

Is it possible to share success_map between threads in this example?

Related, but not an answer: Python Multiprocessing appending list

Upvotes: 1

Views: 714

Answers (1)

Roland Smith
Roland Smith

Reputation: 43563

Multiprocessing uses separate processes, not threads. These processes do not share all memory like threads do.

To share data between processes you could use e.g. multiprocessing.Value or multiprocessing.Array. But be aware that in some cicrumstances you'll need an extra Lock.

Using a Manager object you can share most data types (it must be pickle-able, I think). And it's slower that shared memory.

Or you could create a multiprocessing.Pipe to exchange data between processes.

Notes:

  1. The multiprocessing module does use threads internally for housekeeping.
  2. In general it is better to avoid sending large pieces of data between processes.

Upvotes: 4

Related Questions