CODEWITHSUNDEEP
python-3.xconcurrent.futures
gdogg371
gdogg371

Reputation: 4152

Python 3 concurrent.futures - process for loop in parallel

If I have understood correctly how the concurrent.futures module in Python 3 works, the following code:

import concurrent.futures
import threading

# Simple function returning a value
def test(i):

    a = 'Hello World\n'
    return a


def main():
    output1 = list()

    with concurrent.futures.ThreadPoolExecutor() as executor:

        # psdd iterator to test function
        for out1 in executor.map(test, range(0, 10)):
            # append returned result
            output1.append(out1)

            # confirm output
            print(output1)
            print("Task Executed {}".format(threading.current_thread()))


if __name__ == '__main__':
    main()

...performs the following functions:

  1. Passes a for loop to the function called test().
  2. Processes the loop in parallel, rather than in serial.

However, what I really want is to process the loop in parallel in my main() function as so:

import concurrent.futures
import threading


def main():
    output1 = list()

    with concurrent.futures.ThreadPoolExecutor() as executor:

        # psdd iterator to test function
        for out1 in executor.submit(range(0, 10)):

            a = 'Hello World\n'
            # append returned result
            output1.append(a)

            # confirm output
            print(output1)
            print("Task Executed {}".format(threading.current_thread()))


if __name__ == '__main__':
    main()

...this however produces the following error:

Traceback (most recent call last):
  File "G:\HTPC Scripts\WebGrab Plus\TESTTESTTEST2.py", line 221, in <module>
    main()
  File "G:\HTPC Scripts\WebGrab Plus\TESTTESTTEST2.py", line 209, in main
    for out1 in executor.submit(range(0, 10)):
TypeError: 'Future' object is not iterable

What do I need to amend in my code?

Thanks

Upvotes: 2

Views: 14754

Answers (1)

Roland Smith
Roland Smith

Reputation: 43573

It doesn't matter whether you use submit or map, you always have to use a callable (such as a function) as the first argument.

Python does allow nested functions (also take note of the way to use Futures);

import concurrent.futures


def main():

    def worker(arg):
        return str(arg) + ' Hello World!'

    with concurrent.futures.ThreadPoolExecutor() as e:
        fut = [e.submit(worker, i) for i in range(10)]
        for r in concurrent.futures.as_completed(fut):
            print(r.result())


if __name__ == '__main__':
    main()

The only way to define a callable in-place is with a lambda expression, but those have significant limitations.

Upvotes: 6

Related Questions