CODEWITHSUNDEEP
pythonpython-asyncioaiohttp
zl2003cn
zl2003cn

Reputation: 485

Why asyncio make request with aiohttp still make use of thread

I thought ayncio and the use of coroutine is not related with thread, since coroutine is a type of "thread" running under program's scheduler, so there should be only 1 thread running each process. But when I ran examples in Making 1 million requests with python-aiohttp, the code is like below:

# modified fetch function with semaphore
import random
import asyncio
from aiohttp import ClientSession

async def fetch(url, session):
    async with session.get(url) as response:
        delay = response.headers.get("DELAY")
        date = response.headers.get("DATE")
        print("{}:{} with delay {}".format(date, response.url, delay))
        return await response.read()


async def bound_fetch(sem, url, session):
    # Getter function with semaphore.
    async with sem:
        await fetch(url, session)


async def run(r):
    url = "http://localhost:8080/{}"
    tasks = []
    # create instance of Semaphore
    sem = asyncio.Semaphore(1000)

    # Create client session that will ensure we dont open new connection
    # per each request.
    async with ClientSession() as session:
        for i in range(r):
            # pass Semaphore and session to every GET request
            task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session))
            tasks.append(task)

        responses = asyncio.gather(*tasks)
        await responses

number = 10000
loop = asyncio.get_event_loop()

future = asyncio.ensure_future(run(number))
loop.run_until_complete(future)

With Windows' Resource Monitor, I found that the code create 25 threads in 1 process.

Upvotes: 4

Views: 2549

Answers (2)

hruske
hruske

Reputation: 2253

The aiohttp library uses threads for concurrent DNS resolving by default in order not to block IO loop, see aiohttp/resolver.py. If you want asynchronous DNS lookups, you need to install python package aiodns, which in turn uses pycares.

You can then do:

async def fetch(url):
    resolver = aiohttp.AsyncResolver()
    connector = aiohttp.TCPConnector(resolver=resolver, family=socket.AF_INET)
    async with aiohttp.ClientSession(connector=connector) as session:
        async with session.get(url) as resp:
            if resp.status == 200:
                print("success!")

If you want to set AsyncResolver as a global default, this worked for me with aiohttp 2.2.3:

import aiohttp.resolver
aiohttp.resolver.DefaultResolver = aiohttp.resolver.AsyncResolver

Upvotes: 6

Udi
Udi

Reputation: 30472

Python's standard library includes a module called threading which allows to run python code concurrently by using Thread instances. asyncio and aiohttp does not use the threading module to operate.

Python itself might use OS (low level) threads as an implementation detail - but this probably changes between different platforms and versions. For example, the number of OS threads for a simple print('hello world'); s = input() for python 3.6.0 in Windows 10 is 3.

Check out https://github.com/python/cpython/blob/3.6/Lib/asyncio/windows_events.py to find clues for what might start OS threads in windows.

Upvotes: 1

Related Questions