CODEWITHSUNDEEP
pythonaiohttp
Aliaksei Lithium
Aliaksei Lithium

Reputation: 384

AIOHTTP client timeout due to User-Agent issue

Got trouble with aiohttp client.
The issue: I need to download image via https url, it works great with requests.get() but failed due to timeout with aiohttp.

Here is example that fail:

url = "https://www.miamiherald.com/wps/source/images/miamiherald/facebook.jpg"
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}

async with aiohttp.ClientSession().get(url, headers=headers) as response:
    content = await response.read()

Got: 

Fatal read error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x10fd8be80>
transport: <_SelectorSocketTransport fd=11 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 801, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
TimeoutError: [Errno 60] Operation timed out

In the same time, requests works fine with the same headers!

url = "https://www.miamiherald.com/wps/source/images/miamiherald/facebook.jpg"
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}

r = requests.get(url, headers = headers, stream=True)

Can anyone help me to make it work for this particular example?

Upvotes: 3

Views: 6868

Answers (1)

Yurii Kramarenko
Yurii Kramarenko

Reputation: 1054

Just pass timeout arg to cs.get(timeout=...) or to cs(timeout=...). Here is documentation https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts

Example:

import asyncio
import aiohttp
from aiohttp.client import ClientTimeout


async def test():
    url = "https://www.google.com/photos/about/static/images/google.svg"
    headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
    }

    timeout = ClientTimeout(total=0)  # `0` value to disable timeout
    cs = aiohttp.ClientSession()

    async with cs.request('get', url, headers=headers, timeout=timeout) as response:
        print(response.status)
        content = await response.read()

    await cs.close()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test())

Upvotes: 2

Related Questions