CODEWITHSUNDEEP
pythonmultithreadingasynchronousmultiprocessing
Filip Lav Maksimovic
Filip Lav Maksimovic

Reputation: 103

Running multiple processses asynchronously

So I have a main() function which needs to continously start >20 different processes every 5 seconds without waiting them to finish as some might take over 5 minutes to finish. I’ve used threads so far (without joining them) but GIL eventually halts the execution as number of active threads increases. Could I maybe use some sort of async multiprocessing?

Upvotes: 0

Views: 79

Answers (1)

u1234x1234
u1234x1234

Reputation: 2510

You can try to use concurrent.future.ProcessPoolExecutor to asynchronously run functions in multiple processes without overloading your pc:

import time
from concurrent.futures import ProcessPoolExecutor


def background_func(task):
    time.sleep(task)
    return task


with ProcessPoolExecutor(max_workers=10) as pool:
    for task in range(30):
        r = pool.submit(background_func, task) # non-blocking operation
        r.add_done_callback(lambda x: print(f"completed: {x.result()}"))

Upvotes: 1

Related Questions