CODEWITHSUNDEEP
pythonapiconcurrencypython-asyncioaiohttp
tyu46
tyu46

Reputation: 71

Understanding & debugging `asyncio.TimeoutError from None` error

I am getting an issue with aiohttp, where I get below error but not sure the best way to fix it:

Traceback (most recent call last):
  File "/app/app/services/file_ingestion_utils.py", line 110, in send_api_request
    async with session.post(url, headers=self.headers, data=payload) as response:
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 544, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 905, in start
    self._continue = None
  File "/usr/local/lib/python3.8/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError

I see from here that these errors can be a bit nebulous, but I at least want to understand what is causing these errors at least in my case. My implementation can be seen below.

    async def async_request(self, df, entity):

        api_request_records = []
        ...some logic to prepare records...


        @backoff.on_exception(backoff.expo, aiohttp.ClientError, max_tries=2)
        async def send_api_request(payload, session):

            url = <some_url>

            try:
                async with session.post(url, headers=self.headers, data=payload) as response:
                      ...some response handling logic...
                    
            except asyncio.TimeoutError:
                self.logger.exception(f"Asyncio TimeoutError on {url} and payload {payload}")

        async with aiohttp.ClientSession() as session:
            await asyncio.gather(
                *[send_api_request(api_request_record, session)) for api_request_record in api_request_records])

Q1: When the length of api_request_records is small, the method works fine but when it is large, I'm more likely to get the TimeoutError. Why?

Q2: Is setting the ClientSession(timeout=...) parameter the secret here since according to here, it might be helpful? However, I feel like this response might be a bit outdated since it's pointed out that a ClientTimeout object is now used instead of int. Relatedly, according to official docs here, the default ClientTimeout seems to already be unlimited time via total=None so does this apply for ClientSession as well?

Overall, would love some help and recommended approach. Thanks!

Upvotes: 6

Views: 4468

Answers (1)

2474101468
2474101468

Reputation: 402

From https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts

By default aiohttp uses a total 300 seconds (5min) timeout, it means that the whole operation should finish in 5 minutes.

Therefore,

session_timeout = aiohttp.ClientTimeout(total=None)
session = aiohttp.ClientSession(timeout=session_timeout)

Upvotes: 8

Related Questions