CODEWITHSUNDEEP
pythonpython-3.xmultithreadingmultiprocessingpython-3.7
matth
matth

Reputation: 33

why doesn't multiprocessing use all my cores

So I made a program that calculates primes to test what the difference is between using multithreading or just using single thread. I read that multiprocessing bypasses the GIL, so I expected a decent performance boost.

So here we have my code to test it:

def prime(n):
    if n == 2:
        return n
    if n & 1 == 0:
        return None
    
    d= 3
    while d * d <= n:
        if n % d == 0:
            return None
        d= d + 2
    return n

loop = range(2,1000000)
chunks = range(1,1000000,1000)
def chunker(chunk):
    ret = []
    r2 = chunk + 1000
    r1 = chunk
    for k in range(r1,r2):
        ret.append(prime(k))
    return ret
    
from multiprocessing import cpu_count
from multiprocessing.dummy import Pool
from time import time as t
pool = Pool(12)
start = t()
results = pool.map(prime, loop)
print(t() - start)
pool.close()
filtered = filter(lambda score: score != None, results)


new = []
start = t()
for i in loop:
    new.append(prime(i))
print(t()-start)
pool = Pool(12)
start = t()
results = pool.map_async(chunker, chunks).get()
print(t() - start)
pool.close()

I executed the program and this where the times:

multi processing without chunks:
4.953783750534058
single thread:
5.067057371139526
multiprocessing with chunks:
5.041667222976685

Maybe you already notice, but multiprocessing isn't that much faster. I have a 6 core 12 thread AMD ryzen CPU, so I excpected if I can use all those threads, that I would at least double the performance. But no. If I look in task manager the cpu usage on average from using multiprocessing is 12%, while single threaded uses around 10% of the cpu.

So what is going on? Did I do something wrong? Or does meaning being able to bypass the GIL not mean being able to use more cores? If I can't use more cores with multiprocessing how can I do it then?

Upvotes: 2

Views: 547

Answers (1)

ForceBru
ForceBru

Reputation: 44828

from multiprocessing.dummy import Pool
from time import time as t
pool = Pool(12)

From the documentation:

multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.

In other words, you're still using threads, not processes.

To use processes, do from multiprocessing import Pool instead.

Upvotes: 2

Related Questions