What is the best way to multiprocess for loops?

Question

I am learning the basics of multiprocessing and want to divide for loops into seperate processes to be processed on different cpu cores. I am planning to implement multiprocessing in an existing script, that does heavy computation.

Right now my approach looks like this:

import multiprocessing as mp
from functools import partial

def function(a, b, numbers):
    return a*b

if __name__ == '__main__':
    a = 1
    b = 2
    numbers = range(1000)
    func_part = partial(function, a, b)
    # Pool function
    result = mp.Pool().map(func_part, numbers)
    print(result)

    # equivalent for loop
    L = []
    for i in numbers:
        L.append(function(a, b, i))
    print(L)

1. Is there a better approach in doing this?

2. Is it possible to get the iterator numbers as first parameter of the function without breaking it? numbers has to be passed lastly by the map function it seems.

Answer 1

you can use a context manager:

with mp.Pool() as p:
    print(p.map(func_part, numbers))

---

Is it possible to get the iterator numbers as first parameter of the function without breaking it?

yes, it is possible, but you are not passing the iterator to the function, you are passing an item from your numabers variable

def function(number, a, b, ):
    return a*b

a = 1
b = 2
numbers = range(1000)
func_part = partial(function, a=a, b=b)



with mp.Pool() as p:
    print(p.map(func_part, numbers))