CODEWITHSUNDEEP
pythonmultiprocessing
Kevin
Kevin

Reputation: 3239

Accessing a shared array from multiple processors

The issue:

I want to access the y shared variable across all processes because for each element in new_y I want to make it the current y and the next y something like new_y[i] = y[i] + y[i+1]. How do I get the index that the current pool worker is at of the array y that I sent in?

import multiprocessing

num_processes = 2
y = multiprocessing.Array('d', 6, lock=False)
new_y = multiprocessing.Array('d', 6, lock=False)

def init_process(y_to_share, new_y_to_share):
    global y, new_y
    y = y_to_share
    new_y = new_y_to_share

process_pool = multiprocessing.Pool(
                   num_processes,
                   initializer=init_process,
                   initargs=(y, new_y))

dt = 0.0001
def sq():
  global y
  global new_y
  print new_y[0]
  print multiprocessing.current_process()
  #Here I want to do y at the current index and add the y value of the next index
  #something like new_y[i] = y[i]+y[i+1]


process_pool.map(sq, y)

Upvotes: 0

Views: 51

Answers (1)

tdelaney
tdelaney

Reputation: 77347

I'm hesitant to answer because I may be completely misunderstanding the problem, but you could deal with this by changing what you are iterating in the parent process so that it has the adjacent data you want.

import multiprocessing

def worker(yvals):
    return yvals[0] + yvals[1]

if __name__ == "__main__":
    y_list = list(range(6))
    pool = multiprocessing.Pool()
    new_y = list(pool.map(worker, 
        (y_list[i:i+2] for i in range(len(y_list)-1))))
    pool.close()
    print(new_y)

On linux, when a pool starts it has a copy-on-write view of the parent address space and can just read the list. I'm not sure what happens in this case on Windows but it tries to pickle the parent environment and initialize the child with it - making me wonder why anybody would use this module on Windows! - but for linux and osx at least, this would work

import multiprocessing

def worker(y_index):
    return y_list[y_index] + y_list[y_index+1]

if __name__ == "__main__":
    y_list = list(range(6))
    pool = multiprocessing.Pool()
    new_y = list(pool.map(worker, range(len(y_list)-1)))
    print(new_y)

Upvotes: 1

Related Questions