CODEWITHSUNDEEP
pythonmultiprocessing
dudu
dudu

Reputation: 849

python multiprocessing: processes don't work

I want to assign my computing jobs to more than one cpu, so I choose multiprocessing. However, the result is not what I want.

import numpy as np
from multiprocessing import Process

def func(begin,end):
    print('*'*5)
    print('begin=%d' %(begin))
    for i in range(begin,end):
        for j in range(10):
            myarray[i][j]=1

myarray=np.zeros((12,10))
print(myarray)
for i in range(4):
    begin=i*3
    end=(i+1)*3
    p=Process(target=func,args=(begin,end,))
    p.start()

print('*'*5)
print(myarray)

I think that myarray should be all ones. But it doesn't change at all. Why? The func function does not change elements of myarray? I tried the example from this linkenter link description here

from multiprocessing import Process
def f(name):
    print('hello',name)

p=Process(target=f,args=('bob',))
p.start()

It shows nothing on the screen. Why? How should I finish my computation with python? Can anyone give a way of fully take advantage of multi-cpus?

Upvotes: 3

Views: 1726

Answers (2)

peihuadeng
peihuadeng

Reputation: 1

That is because the print(myarray) statement execute before the process end. You should wait for the process end and then execute this statement. Please note the thread.join() statement at the following.

import numpy as np
import threading

lock = threading.RLock()
thread_list = []
def func(begin,end):
    print('*'*5)
    print('begin=%d' %(begin))
    for i in range(begin,end):
        for j in range(10):
            with lock:
                myarray[i][j]=1

myarray=np.zeros((12,10))
print(myarray)
for i in range(4):
    begin=i*3
    end=(i+1)*3
    p=threading.Thread(target=func,args=(begin,end,))
    p.start()
    thread_list.append(p)

print('*'*5)
for thread in thread_list:
    thread.join()
print(myarray)

Upvotes: 0

DWilches
DWilches

Reputation: 23016

There are two problems there:

  1. When you print your array at the end, how do you know your processes have already finished? You need to invoke join() on each process to ensure they have finished.

  2. Each process has a copy of "myarray". If you want to communicate several processes you need to either use a Queue or a Pipe. Check the documentation that talks about exchanging data between processes

Here is a working example using as base the one you posted (it is not intended to be fast, just to show how the communication is done):

from multiprocessing import Process, freeze_support, Queue

def func(my_id, q, begin, end):
    global myarray
    print('Process %d has range: %d - %d' % (my_id, begin, end))
    for i in range(begin,end):
        q.put((i, i * 2))

if __name__ == "__main__":
    freeze_support()

    q = Queue()
    processes = []
    myarray=[0] * 12
    print("At the beginning the array is ", myarray)

    for i in range(4):
        begin = i*3
        end = (i+1)*3
        p = Process(target=func, args=(i, q, begin, end))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

    while not q.empty():
        (index, value) = q.get()
        myarray[index] = value

    print("At the end the array is ", myarray)

Try to change the line p.join() by pass to see what happens :)

Upvotes: 2

Related Questions