CODEWITHSUNDEEP
pythonmultiprocessing
Damao
Damao

Reputation: 35

Python:Why the manager dict() in pool shows the variable is not defined in multiprocessing?

My code is this:

from multiprocessing import Process, Manager,Pool
station=["A","B","C"]   
def test(k):
    try:
        print(phicc)
    except Exception as E:
        print(E)
        print(station[k])
        


if __name__ == '__main__':
    
      with Manager() as manager:
        phicc=manager.dict()
        for i in station:
           
           phicc[i]=manager.list()
           
        pool = manager.Pool(processes = 10)
        pool.map_async(test,range(len(station)))
        pool.close()
        pool.join()

Then the output is

name 'phicc' is not defined
A
name 'phicc' is not defined
B
name 'phicc' is not defined
C

I don't know what happened! I need the phicc variable could be recognized in the test function! Thanks

Upvotes: 1

Views: 261

Answers (1)

Booboo
Booboo

Reputation: 44223

You need to initialize the global variable phicc for each process in your multiprocessing pool and the way to do that is by using the initializer and initargs arguments of the multiprocessing.pool.Pool constructor:

from multiprocessing import Process, Manager, Pool


def init_pool_processes(d):
    global phicc

    phicc = d

station = ["A", "B", "C"]

def test(k):
    try:
        print(phicc)
    except Exception as E:
        print(E)
        print(station[k])


if __name__ == '__main__':

    with Manager() as manager:
        phicc = manager.dict()
        for i in station:
            phicc[i] = manager.list()

        pool = manager.Pool(processes=10, initializer=init_pool_processes, initargs=(phicc,))
        pool.map_async(test, range(len(station)))
        pool.close()
        pool.join()

Prints:

{'A': <ListProxy object, typeid 'list' at 0x1e2df934eb0>, 'B': <ListProxy object, typeid 'list' at 0x1e2df940130>, 'C': <ListProxy object, typeid 'list' at 0x1e2df940310>}
{'A': <ListProxy object, typeid 'list' at 0x1e2df934eb0>, 'B': <ListProxy object, typeid 'list' at 0x1e2df940130>, 'C': <ListProxy object, typeid 'list' at 0x1e2df940310>}
{'A': <ListProxy object, typeid 'list' at 0x1e2df934eb0>, 'B': <ListProxy object, typeid 'list' at 0x1e2df940130>, 'C': <ListProxy object, typeid 'list' at 0x1e2df940310>}

Notes

I find it a bit odd that you are using a Pool proxy created by the manager.Pool call instead of creating a multiprocessing.pool.Pool class since the code you posted clearly has no need for that. If you use the "standard" pool and if you were running under Linux or another platform that used OS fork to create new processes, then strictly speaking you would not have to use a pool initializer as in the above example for each new process created would inherit the global variables of the main process (they are, however, not "sharable", that is if the subprocess modifies the global variable it will be a change to its own local copy). So the following code would work under Linux:

from multiprocessing import Process, Manager, Pool


station = ["A", "B", "C"]

def test(k):
    try:
        print(phicc)
    except Exception as E:
        print(E)
        print(station[k])


if __name__ == '__main__':

    with Manager() as manager:
        phicc = manager.dict()
        for i in station:
            phicc[i] = manager.list()

        # A multiprocessing.pool.Pool instance:
        pool = Pool(processes=10)
        pool.map_async(test, range(len(station)))
        pool.close()
        pool.join()

Prints:

{'A': <ListProxy object, typeid 'list' at 0x7f8c08a663d0>, 'B': <ListProxy object, typeid 'list' at 0x7f8c08a664c0>, 'C': <ListProxy object, typeid 'list' at 0x7f8c08a662e0>}
{'A': <ListProxy object, typeid 'list' at 0x7f8c08a663d0>, 'B': <ListProxy object, typeid 'list' at 0x7f8c08a664c0>, 'C': <ListProxy object, typeid 'list' at 0x7f8c08a662e0>}
{'A': <ListProxy object, typeid 'list' at 0x7f8c08a663d0>, 'B': <ListProxy object, typeid 'list' at 0x7f8c08a664c0>, 'C': <ListProxy object, typeid 'list' at 0x7f8c08a662e0>}

This is why when posting a question tagged with multiprocessing you are also requested to tag your question with your platform, such as windows or linux, for there are different solutions possible. The code posted that uses a pool initializer will work for both Windows and Linux but the above Linux-only code would be more efficient if your platform is Linux.

Upvotes: 1

Related Questions