CODEWITHSUNDEEP
pythonpython-3.xpython-multiprocessing
mmg
mmg

Reputation: 11

Share the list of lists in multiprocessing

I want to increase the efficiency of my code. One intensive part of my code is to append elements to a list of lists. Basically, I want to do something as follows,

import multiprocessing
import time
def update_val(L, i):  
    L.append(i**2)
    return L

if __name__ == "__main__":
    N = 1000000
    x_reg = [list(range(10)) for i in range(N)]
    y_reg = [list(range(10)) for i in range(N)]
    z_reg = [list(range(10)) for i in range(N)]   
    "Regular Call"
    start = time.time()
    [x_reg[i].append(i**2)  for i in range(N)]
    stat_reg =time.time() - start

    "Multiprocessing"
    num_cores = multiprocessing.cpu_count() # equals 4 in my case
    pool = multiprocessing.Pool(num_cores)
    start = time.time()
    y_reg = pool.starmap(update_val,[(y_reg[i],i) for i in range(N)])
    pool.close()
    pool.join()
    stat_val =time.time() - start


    print("Regular: %g "%(stat_reg))
    print("Mult.: %g "%(stat_val))

The output is:

Regular: 0.387984 
Mult.: 2.54244

I believe the reason is related to how multiprocessing works; It needs to make a copy of the original list, do staff and return it. This should be the main reason why multiprocessing is very slow in my case. Here is my question: 1- How should I perform the same function to z_reg while sharing it between processes? 2- Does it enhance the performance. 3- Any other idea how to attach something to every sub-list in z_reg?

Upvotes: 1

Views: 1601

Answers (1)

nosklo
nosklo

Reputation: 223122

I suggest you use a multiprocessing.Queue. Keep the lists in the main process, and send the data from all children processes through the Queue.

Upvotes: 1

Related Questions