CODEWITHSUNDEEP
pythonpython-asyncioaiohttp
Dmitry Tolkach
Dmitry Tolkach

Reputation: 51

I can't use aiohttp session if previous query failed with timeout

I have a simple script that create async tasks for loading different pages. The first request fails with TimeoutError, and it causes next queries to fail too. But the second one has much longer timeout and should pass.

Is it possible let other queries don't fail?

import aiohttp
import asyncio
import logging


logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)s %(levelname)s %(message)s')


async def main():
    asyncio.ensure_future(
        load_page('https://www.netflix.com:5000'))

    asyncio.ensure_future(
        load_page('http://bash.im/', 10))

    asyncio.ensure_future(
        load_page('https://myshows.me/'))

    asyncio.ensure_future(
        load_page('http://www.lostfilm.tv/'))


async def load_page(url, timeout=3):
    try:
        async with session.get(url, timeout=timeout) as response:
            text = await response.text()
            print(len(text))

    except Exception:
        logging.warning(type(e))


if __name__ == '__main__':
    loop = asyncio.get_event_loop()

    conn = aiohttp.TCPConnector(limit=1)
    session = aiohttp.ClientSession(connector=conn, loop=loop)

    asyncio.ensure_future(main())
    loop.run_forever()

Log:

2017-06-26 13:57:37,869 asyncio DEBUG Using selector: EpollSelector
2017-06-26 13:57:41,780 root WARNING <class 'concurrent.futures._base.TimeoutError'>
2017-06-26 13:57:41,780 root WARNING <class 'concurrent.futures._base.TimeoutError'>
2017-06-26 13:57:41,780 root WARNING <class 'concurrent.futures._base.TimeoutError'>
2017-06-26 13:57:48,780 root WARNING <class 'concurrent.futures._base.TimeoutError'>

Upvotes: 4

Views: 1102

Answers (1)

Yuval Pruss
Yuval Pruss

Reputation: 9856

Yes, It's possible. I rewrite your code and use some of this concept to fill your's requests:

  • I used itertools.starmap in order to pass multiple arguments and create a list of each arguments which passed to the load_page function.

  • I used asyncio.gather in order to wrap the tasks together, and used changed the return_exceptions flag to true to ensure that the exceptions won't raised.

  • Changed the async before the def main. It returns the gathered tasks right now.

  • I closed the loop in the end.

The code:

import aiohttp
import asyncio
import logging
import itertools

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)s %(levelname)s %(message)s')

def main(session):
    args = [
        ('https://www.netflix.com:5000', session,),
        ('http://bash.im/', session, 10),
        ('https://myshows.me/', session,),
        ('http://www.lostfilm.tv/', session,),
    ]
    tasks = itertools.starmap(load_page, args)
    futures = map(asyncio.ensure_future, tasks)

    return asyncio.gather(*futures, return_exceptions=True)


async def load_page(url, session, timeout=3):
    try:
        async with session.get(url, timeout=timeout) as response:
            text = await response.text()
            print(len(text))
    except Exception:
        logging.warning(type(e))


if __name__ == '__main__':
    loop = asyncio.get_event_loop()

    conn = aiohttp.TCPConnector(limit=1)
    session = aiohttp.ClientSession(connector=conn, loop=loop)
    loop.run_until_complete(main(session))
    loop.close()

For more reading about: asyncio.gather.

For more reading about: itertools.starmap.

Enjoy!

Upvotes: 1

Related Questions