CODEWITHSUNDEEP
pythonpython-3.xmultithreadingmultiprocessing
DachuanZhao
DachuanZhao

Reputation: 1339

Why does the application can run faster when the number of processes is greater than the total number of threads in CPU in python?

Why does the application run faster by using more processes when the number of processes is greater than the total number of threads in CPU in python? Here is the code

import time
from concurrent.futures import ProcessPoolExecutor

def wait():
    time.sleep(1)

start = time.time()

#create the process pool
with ProcessPoolExecutor(max_workers=40) as executor:
    future = []
    for _ in range(400):
        future.append(executor.submit(wait))
    start2 = time.time()
    future_res = []
    for i,v in enumerate(future):
        future_res.append(v.result())
    print(time.time()-start)
    print(time.time()-start2)

its output is:

10.364367246627808
9.99793791770935

Does it mean that a application will run faster and faster if I use more and more processes?

Upvotes: 2

Views: 125

Answers (1)

J_H
J_H

Reputation: 20450

The given wait function doesn't consume any interesting amount of CPU cycles; it only interacts with the scheduler. It is hardly surprising that higher concurrency levels would let lots of overlapping sleep's complete quicker. Here you chose num_tasks = 400. For large values, your completion time will tend toward the ratio of num_tasks / max_workers. This is absolutely not the behavior that a CPU-bound workload would experience.

If you have a different workload in mind as your "real" problem, ask about it in a separate question, and describe whether it is largely CPU-, I/O-, or network-bound.

Upvotes: 2

Related Questions