CODEWITHSUNDEEP
python-3.xpython-asyncioaiohttp
Daniel
Daniel

Reputation: 3531

Why does the json() method of an aiohttp response require await?

I do not understand why resp.json() needs to be awaited. From my understanding async/await is useful when dealing with I/O. But when I call resp.json() in the example below, has the web request not already been processed with session.get() in the line above? 

async with session.get('https://api.github.com/events') as resp:
    print(await resp.json())

Upvotes: 4

Views: 3887

Answers (2)

Scarface97
Scarface97

Reputation: 103

I went down the rabbit hole with this, so let me explain.

You are ofc right in that a conversion to JSON is not I/O bound, and it's hardly CPU-bound either (unless the response body is really big).

However, in the example of aiohttp, it's not the conversion to JSON that is being awaited, although it misleadingly appears so in the provided interface.

What aiohttp does is, even though the server returns everything (the status code, headers and body) in one go (assuming a non-streaming situation, that's just how HTTP works), the client can buffer all of it into memory, and read it in steps. So, it can read just the status code first, and depending on its value, it can proceed or not with the other steps. Each step is, technically speaking, a separate I/O operation, even though we generally don't consider reading from memory as a bottleneck, since it's super fast.

Also, if the response body is huge, it may have not fully arrived yet, so the extra step would involve network I/O too.

This separation is quite useful for large responses, but mostly just adds asyncio overhead otherwise. Remember that async programming should be used only where it makes sense, if not every function would be an awaitable by default and we wouldn't have async/await keywords.

It's a baked in decision in aiohttp though, so there's not much that you can do. Ultimately, you gain more than what you lose here overall (compared to using requests throughout your project) so you should probably be fine with the trade-off.

Upvotes: 1

Mikhail Gerasimov
Mikhail Gerasimov

Reputation: 39576

But when I call resp.json() in the example below, has the web request not already been processed with session.get() in the line above?

No, it reads only HTTP headers, to get response body you need to read the rest of the response.

It's pretty useful since you can check HTTP headers and avoid reading the rest of the response if, let's say, server returned wrong HTTP code.

Another example: if you expect response body to be big, you can read it by chunks to avoid RAM overusage (check note here).

Upvotes: 5

Related Questions