CODEWITHSUNDEEP
pythonloopsnumpymultiprocessingnested-loops
mathguy
mathguy

Reputation: 1518

Multiprocessing with nested loops and some numpy function calls

I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:

import numpy as np

def data_processing(x,y,z): return np.array([x,y])*(z**0.5)

def foo(n1,n2):

    final_result = {}

    for i in range(n1):
        result = np.zeros([n2,n2])

        for j1 in range(n2):
            for j2 in range(j1):
                temp= data_processing(j1,j2,i)
                result[j1,j2] = np.prod(temp)

        final_result[str(i)] = result

    return final_result

if __name__ == '__main__':
    X = foo(9,9)

If I want to run this piece of code while utilizing all of the cpu cores, what should I change?

Upvotes: 1

Views: 92

Answers (1)

apatrck00
apatrck00

Reputation: 56

Maby this can help. 

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
    final_result = {}

    for i in range(n1):
        result = np.zeros([n2, n2])

        for j1 in range(n2):
            for j2 in range(j1):
                temp = data_processing(j1, j2, i)
                result[j1, j2] = np.prod(temp)

        final_result[str(i)] = result

    return_dict[id] = final_result



stamp = time.time()
def pint(num):
    print(f'*Test [{num}] - seconds: {time.time() - stamp}')

for i in range(10):
    foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
    p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
    processes.append(p)
    p.start()

for p in processes:
    p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

My output is:

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

Upvotes: 1

Related Questions