CODEWITHSUNDEEP
pythonpython-3.xconcurrencypython-multithreading
chhantyal
chhantyal

Reputation: 12252

Why does concurrent.futures executor map throw error when using with futures.as_completed after all the futures are complete?

I am trying to send HTTP requests concurrently. In order to do so, I am using concurrent.futures

Here is simple code:

import requests
from concurrent import futures

data = range(10)


def send_request(item):
    requests.get("https://httpbin.org/ip")
    print("Request {} complete.".format(item))


executor = futures.ThreadPoolExecutor(max_workers=25)

futures_ = executor.map(send_request, data)

for f in futures.as_completed(futures_):
    f.result()

If I run it, I can see requests are sent asynchronously, which is exactly what I want to do. However, when all the requests are complete, I get following error:

Request 0 complete.
Request 6 complete.
...
Request 7 complete.
Request 9 complete.
Request 3 complete.
Traceback (most recent call last):
  File "send_thread.py", line 18, in <module>
    for f in futures.as_completed(futures_):
  File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 219, in as_completed
    with _AcquireFutures(fs):
  File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 146, in __enter__
    future._condition.acquire()
AttributeError: 'NoneType' object has no attribute '_condition'

This is quite strange error. Here executor.map seems to be problematic. If I replace map with following line, it works as expected.

futures_ = [executor.submit(send_request, x) for x in data]

What am I missing? Tried to find difference between two, but can't seem to understand what could cause above issue. Any input would be highly appreciated.

Upvotes: 5

Views: 7791

Answers (1)

skovorodkin
skovorodkin

Reputation: 10274

Executor.map does not return you a list of futures but a generator of results, so instead of:

futures_ = executor.map(send_request, data)
for f in futures.as_completed(futures_):
    f.result()

you should run:

results = executor.map(send_request, data)
for r in results:
    print(r)

Upvotes: 14

Related Questions