CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingpython-multiprocessing
newbie
newbie

Reputation: 1412

How to have a global/common variable in python multiprocessing

I have recently started using multiprocessing in python and I have the following code to update the list items from multiple processes. But it is giving empty list.

from multiprocessing import Pool
import time

global_list = list()


def testfun(n):
    print('started ', n)
    time.sleep(1)
    global_list.append(n)
    print('completed ', n)


def call_multiprocessing_function():
    mytasks = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n']
    with Pool() as pool:
        pool.map(testfun, mytasks)


if __name__ == "__main__":
    print('starting the script')

    print(global_list)
    call_multiprocessing_function()
    print(global_list)

    print('completed the script')

I am getting the following output

starting the script
[]
started  a
started  b
started  c
started  d
completed  a
started  e
completed  b
started  f
completed  c
started  g
completed  d
started  h
completed  e
started  i
completed  f
started  j
completed  g
started  k
completed  h
started  l
completed  i
started  m
completed  j
started  n
completed  k
completed  l
completed  m
completed  n
[]
completed the script

The result list is coming as empty. Is there a way to have a common variable to be shared across all these processes to store the data. How can we achieve this functionality using multiprocessing?

Upvotes: 2

Views: 192

Answers (1)

vks
vks

Reputation: 67988

Processes do not share memory.So you need to use Manager.list

import time    
from multiprocessing import Pool, Manager

m=Manager()
global_list = m.list()


def testfun(n):
    print('started ', n)
    time.sleep(1)
    global_list.append(n)
    print('completed ', n)


def call_multiprocessing_function():
    mytasks = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n']
    p=Pool()
    p.map(testfun, mytasks)


if __name__ == "__main__":
    print('starting the script')

    print(global_list)
    call_multiprocessing_function()
    print(global_list)

    print('completed the script')

Output:

starting the script
[]
started  a
started  b
started  c
started  d
started  e
started  f
started  g
started  h
completed  e
started  i
completed  f
started  j
completed  d
started  k
completed  a
started  l
completed  g
started  m
completed  b
completed  c
started  n
completed  h
completed  i
completed  j
completed  k
completed  l
completed  n
completed  m
['e', 'f', 'd', 'a', 'g', 'b', 'c', 'h', 'i', 'j', 'k', 'l', 'n', 'm']
completed the script

Upvotes: 5

Related Questions