CODEWITHSUNDEEP
pythonmultithreading
mesboomin
mesboomin

Reputation: 35

Processing different tasks in threads

I am trying to process different tasks in threads, when a task is completed at a specific thread, it is expected to create a new thread and process the next available task.

I can't manage to do it with the following code.

import threading
results={}

def func(task): #an example function
    results[task]=(task+1)


myThreads=[]
tasks=[i for i in range(2000)] 
while len(items)>0:
    if len(myThreads)<50: #max active thread is 50
        task=tasks.pop() #get an item from tasks
        myThread=threading.Thread(target=func,args=[task])
        myThreads.append(myThread)
        myThread.start()
    else:
        for t in myThreads: #checking all threads
            if not t.is_alive(): #if the thread(task) is finished
                myThreads.remove(t) #remove the thread from list
                del t #this might not be necessary

for my_thread in myThreads:
    my_thread.join()

Upvotes: 0

Views: 131

Answers (1)

HTF
HTF

Reputation: 7260

It sounds that what you need is a pool:

import time
  
from concurrent.futures import ThreadPoolExecutor


def worker(i):
    time.sleep(5)
    return i * 2


def main():
    tasks = range(10)

    with ThreadPoolExecutor() as executor:
        result = executor.map(worker, tasks)
        print(*result)


if __name__ == '__main__':
    main()

Test:

$ time python test.py 
0 2 4 6 8 10 12 14 16 18

real    0m5.045s
user    0m0.034s
sys 0m0.008s

Upvotes: 1

Related Questions