CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
user1423020
user1423020

Reputation: 275

multiprocessing: variable being referenced before assignment in some cases but not others

I found the following example on this website somewhere:

import multiprocessing
import ctypes
import numpy as np

shared_array_base = multiprocessing.Array(ctypes.c_double, 10*10)
shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
shared_array = shared_array.reshape(10, 10)

# No copy was made
assert shared_array.base.base is shared_array_base.get_obj()

# Parallel processing
def my_func(i, def_param=shared_array):
    shared_array[i,:] = i

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    pool.map(my_func, range(10))

    print shared_array

The above code works fine, but if I want to add an array to the shared array, something like shared_array += some_other_array (instead of the above shared_array[i,;] = i) I get

local variable 'shared_array' referenced before assignment

Any ideas why I cannot do that?

Upvotes: 4

Views: 3923

Answers (1)

BrenBarn
BrenBarn

Reputation: 251568

If a variable is assigned to anywhere in a function, it is treated as a local variable. shared_array += some_other_array is equivalent to shared_array = shared_array + some_other_array. Thus shared_array is treated as a local variable, which does not exist at the time you try to use it on the right-hand side of the assignment.

If you want to use the global shared_array variable, you need to explicitly mark it as global by putting a global shared_array in your function.

The reason you don't see the error with shared_array[i,:] = i is that this does not assign to the variable shared_array. Rather, it mutates that object, assigning to a slice of it. In Python, assigning to a bare name (e.g., shared_array = ...) is very different from any other kind of assignment (e.g., shared_array[...] = ...), even though they look similar.

Note, incidentally, that the error has nothing to do with multiprocessing.

Upvotes: 6

Related Questions