CODEWITHSUNDEEP
pythonpython-3.xdictionaryasynchronouspython-asyncio
captobvious
captobvious

Reputation: 51

Python asyncio gather dictionary

So I understand how to create a collection of async tasks via a list and use asyncio.gather() to execute them but I don't know how to do the same with a dict:

    from fastapi import FastAPI
    import asyncio
    import httpx

    app = FastAPI()

    urls = [
        "http://www.google.com",
        "https://www.example.org",
        "https://stackoverflow.com/",
        "https://www.wikipedio.org"
    ]

    async def async_request_return_dict(url: str):
        async with httpx.AsyncClient() as client:
            r = await client.get(url)
        return {url : r.status_code}

    # List of async coroutines
    @app.get("/async_list")
    async def async_list():

        tasks = []
        for url in urls:
            tasks.append(async_request_return_dict(url=url))
        response = await asyncio.gather(*tasks)

        # Convert list of dict -> dict
        dict_response = dict()
        for url in response:
            dict_response.update(url)

        return dict_response

    async def async_request_return_status(url: str):
        async with httpx.AsyncClient() as client:
            r = await client.get(url)
        return r.status_code

    # Dict of async coroutines
    @app.get("/async_dict")
    async def async_dict():

        tasks = dict()
        for url in urls:
            tasks[url] = async_request_return_status(url=url)

        ### How do you run async co-routines inside a dict??? ###
        await asyncio.gather(*tasks.values())
        
        print(tasks)

        return tasks

Output of /async_list:

    {
        "http://www.google.com":200,
        "https://www.example.org":200,
        "https://stackoverflow.com/":200,
        "https://www.wikipedio.org":200
    }

Traceback for /async_dict :

INFO:     Application startup complete.
{'http://www.google.com': <coroutine object async_request_return_status at 0x7fec83ded6c0>, 'https://www.example.org': <coroutine object async_request_return_status at 0x7fec83ded740>, 'https://stackoverflow.com/': <coroutine object async_request_return_status at 0x7fec83ded7c0>, 'https://www.wikipedio.org': <coroutine object async_request_return_status at 0x7fec83ded840>}
INFO:     127.0.0.1:58350 - "GET /async_dict HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 209, in app
    response_data = await serialize_response(
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/fastapi/routing.py", line 137, in serialize_response
    return jsonable_encoder(response_content)
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/fastapi/encoders.py", line 90, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "/home/cmaggio/repos/async_dict/.venv/lib/python3.8/site-packages/fastapi/encoders.py", line 141, in jsonable_encoder
    raise ValueError(errors)
ValueError: [TypeError("'coroutine' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

Basically how to execute asyncio.gather() with a dict. Or is the first approach of executing the coroutines inside a list and re-constructing a dictionary once all await-ables are resolved.

Upvotes: 5

Views: 10537

Answers (1)

HTF
HTF

Reputation: 7270

This won't work as you expect because your dict values are coroutines but the actual result values are returned from asyncio.gather.

asyncio.gather:

The order of result values corresponds to the order of awaitables in aws.

Based on this fact you can do something like this:

test.py:

import asyncio

import httpx


URLS = (
    "http://www.google.com",
    "https://www.example.org",
    "https://stackoverflow.com/",
    "https://www.wikipedio.org",
)


async def req(url):
    async with httpx.AsyncClient() as client:
        r = await client.get(url)

    return r.status_code


async def main():
    tasks = []

    for url in URLS:
        tasks.append(asyncio.create_task(req(url)))

    results = await asyncio.gather(*tasks)

    print(dict(zip(URLS, results)))


if __name__ == "__main__":
    asyncio.run(main())

Test:

$ python test.py
{'http://www.google.com': 200, 'https://www.example.org': 200, 'https://stackoverflow.com/': 200, 'https://www.wikipedio.org': 200}

Upvotes: 9

Related Questions