CODEWITHSUNDEEP
pythonmultiprocessor
Bobo
Bobo

Reputation: 971

is synchronization needed with multiprocessor in python?

when using a code like this 

def execute_run(list_out): 
   ... do something 

pool = ThreadPoolExecutor(6)
for i in list1:
    for j in list2:
            pool.submit(myfunc, list_out)
pool.join()

assuming the threads modify list_out, do they do that in a synchronous manner?

Upvotes: 1

Views: 886

Answers (3)

Jimilian
Jimilian

Reputation: 3919

If your goal is calculate something in multiprocessing way, it's better do not share state. I propose you to use simple map from multiprocessing if it's possible:

from multiprocessing import Pool

input_list = []
for i in list1:
    for j in list2:
        input_list.append((i, j))

p = Pool()
result_list = p.map(do_something, input_list)

map works like for-loop:

def naive_map(input_list, do_something):
   result = []
   for i in input_list:
     result.append(do_something(i))
   return result

So, if you want to use function that is accepting several arguments, you can use lambda function for unpacking tuple.

  >> def your_function(v1, v2):
  >>        return v1+v2
  >> f = lambda (x,y): your_function(x, y)
  >> map(f, [(1,2),(3,4),(5,6)])
       [3, 7, 11]

Upvotes: 2

Dunes
Dunes

Reputation: 40753

The answer is that each process receives a copy of the list, and so won't see changes made by the other processes.

To achieve what you want you will have to use a Manager to create a list proxy. Note that the manager proxy classes do not know when a member is mutated. For example, if an element of a list proxy is mutated somehow, the list proxy has no way of knowing this. You must reassign the member to flush the changes. An example from the documentation:

# create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append({})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# reassigning the dictionary, the proxy is notified of the change
lproxy[0] = d

Upvotes: 2

tdelaney
tdelaney

Reputation: 77367

multiprocessing thread pools are just threads and don't have any magic to generally synchronize shared objects. You need to protect shared objects with a lock.

Upvotes: 2

Related Questions