CODEWITHSUNDEEP
pythonparallel-processingmultiprocessing
Dr. Who
Dr. Who

Reputation: 141

python multiprocessing pool method is not working

I am setting up a parallel computing pipeline using 50 cores. However, the system is only using 1 core. Here is a test code I am using.

from multiprocessing import Pool
import math

def f(x):
    return math.factorial(x)

if __name__ == '__main__':
    with Pool(50) as p:
        print(p.map(f, range(10000)))

I saved the code in a script and ran it. However, only one core is used. Any suggestions on what did I do wrong?

enter image description here

Upvotes: 0

Views: 81

Answers (1)

Omer Tuchfeld
Omer Tuchfeld

Reputation: 3022

In order to print the p.map(f, range(10000)) expression, str is implicitly called on p.map(f, range(10000)).

Calling str on such a long list with such large numbers is VERY slow. This is what's taking up all the time. The calculation itself is done almost instantly, so you miss it when looking at the core usage.

Note that print is also relatively fast in this case. It's specifically the conversion of the list to a string that's taking up the time here.

Upvotes: 1

Related Questions