CODEWITHSUNDEEP
pythonpython-3.xpython-asyncioaiohttp
TSN
TSN

Reputation: 51

Dealing with aiohttp session.get() timeout issues for large amout of requests

I struggle to find a lot of documentation on aiohttp, specially some recent one. I keep going through the same few pages again and again, github issues and documentation mostly and can't find the key to my problem.

The problem is that while doing a lot of requests, the pending ones timeout before they get to be sent.

This is due to the request's (session.get()) default timeout of 5 minutes which counts from the time the await is done and not the actual request if I understand it well. Thus I am blocked because leaving a fixed timeout will cancel (timeout) all the pending requests after that time or having timeout set to None will make the the program wait indefinitely on the non responding sites. The same problem is raised in this issue but I don't get the workarounds.

Here's my failing code:

async def request(session, url):
    try:
        resp = await session.get(url)
    except:
        "FAIL"
    else:
        "SUCCESS"

async def session():
    urls = loadUrls()
    coros = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            coros.append(request(session, url))
        await asyncio.gather(*coros, return_exceptions=True)

def main():
    loop = asyncio.new_event_loop()
    loop.run_until_complete((session()))
    loop.close()

I've tried, to set the TCPConnector limit=None, with no success.

I saw some people talking about using asyncio.wait instead of asyncio.gather, but as in the issue 3203 shown above, no success.

I've tried to set a ClientSession per each request with no succes but this might be because my lack of understanding.

Thank you for your guidance.

Upvotes: 5

Views: 9756

Answers (1)

Yaakov Bressler
Yaakov Bressler

Reputation: 12168

You can customize the timeout on your ClientSession using ClientTimeout as follows:

my_timeout = aiohttp.ClientTimeout(
    total=None, # total timeout (time consists connection establishment for a new connection or waiting for a free connection from a pool if pool connection limits are exceeded) default value is 5 minutes, set to `None` or `0` for unlimited timeout
    sock_connect=10, # Maximal number of seconds for connecting to a peer for a new connection, not given from a pool. See also connect.
    sock_read=10 # Maximal number of seconds for reading a portion of data from a peer
)

client_args = dict(
    trust_env=True,
    timeout=my_timeout
)

# Pass args
async with aiohttp.ClientSession(**client_args) as session:
    ...
    # do stuff here
    for url in urls:
        coros.append(request(session, url))
    
    # Try awaiting, if a timeout, do something else
    try:
        await asyncio.gather(*coros, return_exceptions=True)
    except asyncio.exceptions.TimeoutError:
        print('Timeout!')

Upvotes: 4

Related Questions