CODEWITHSUNDEEP
pythonmultiprocessing
Randomgenerator
Randomgenerator

Reputation: 83

Using Global Variables in Multiprocessing Pool

I am having issues with trying to use global variables in a multiprocessing pool. I know there is quite a lot of content on this already, but I am struggling to implement the answers to my case.

I have a function that I want to use with pool.map. I want the input of this function to be a few global variables and also one other parameter that changes process-to-process. That is I want something that looks like:


if __name__ == '__main__':

   A = Global Variable Array
   B = Global Variable Array
   C = Actual Input Array For the Multiprocessing

   pool = multiprocessing.Pool()
   Res= pool.map(Function,Input = A,B,C)

In the above case only C varies from process-to-process. Currently my code only works if I import the global arrays from an external files within each process, which seems very wasteful.

Upvotes: 0

Views: 1899

Answers (1)

alex_noname
alex_noname

Reputation: 32283

multiprocessing provides a couple of ways to share state between processes. Access to these objects may be synchronized by lock arg.

If lock is True (the default) then a new recursive lock object is created to synchronize access to the value. If lock is a Lock or RLock object then that will be used to synchronize access to the value. If lock is False then access to the returned object will not be automatically protected by a lock, so it will not necessarily be “process-safe”.

Example from the official documentation:

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

Upvotes: 1

Related Questions