CODEWITHSUNDEEP
pythonmultithreadingnumpymultiprocessing
HighwayJohn
HighwayJohn

Reputation: 921

Multiprocessing nested python loops

To improve my code which has one heavy loop I need a speed up. How can I implement multiprocessing for a code like this? (a is typical of size 2 and l up to 10)

for x1 in range(a**l):
    for x2 in range(a**l):
        for x3 in range(a**l):
            output[x1,x2,x3] = HeavyComputationThatIsThreadSafe1(x1,x2,x3)

Upvotes: 1

Views: 1325

Answers (1)

JoshAdel
JoshAdel

Reputation: 68682

If the HeavyComputationThatIsThreadSafe1 function only uses arrays and not python objects, I would using a concurrent futures (or the python2 backport) ThreadPoolExecutor along with Numba (or cython) with the GIL released. Otherwise use a ProcessPoolExecutor.

See:

http://numba.pydata.org/numba-doc/latest/user/examples.html#multi-threading

You'd want to parallelize the calculation at the level of the outermost loop and and then fill output from the chunks resulting from each thread/process. This assumes the cost of doing so is much cheaper than the computation, which should be the case.

Upvotes: 3

Related Questions