CODEWITHSUNDEEP
pythonmultithreading
Lizbeth
Lizbeth

Reputation: 223

Batching threads in python

I'm updating some existing code that looks like this:

for i in list
  thread.start_new_thread(main, (i[0],i[1],i[2],2))

This causes threads to be created for every item in the list, but not executed until all the threads are created. I'd like to either execute the threads in small groups, or just have them execute directly after they're created.

(There are a lot of python threads discussion on here so sorry if I missed this type of question already being asked...)

Upvotes: 4

Views: 921

Answers (1)

lqc
lqc

Reputation: 7338

You may find the concurrent.futures module useful. It's available also for Python 2 under the name futures. For example, to invoke a function main for every item on MY_LIST concurrently, but with at most 5 threads, you can write:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=5) as executor:
    for result in executor.map(main, MY_LIST):
        # Do something with result
        print(result)

Upvotes: 4

Related Questions