CODEWITHSUNDEEP
python-3.xpython-requestspython-asyncio
devaerial
devaerial

Reputation: 2201

How to run requests.get asynchronously in Python 3 using asyncio?

I'm trying to create simple web monitoring script which sends GET request to urls in list periodically and asynchronously. Here is my request function:

def request(url,timeout=10):
    try:
        response = requests.get(url,timeout=timeout)
        response_time = response.elapsed.total_seconds()
        if response.status_code in (404,500):
            response.raise_for_status()
        html_response = response.text
        soup = BeautifulSoup(html_response,'lxml')
        # process page here
        logger.info("OK {}. Response time: {} seconds".format(url,response_time))
    except requests.exceptions.ConnectionError:
        logger.error('Connection error. {} is down. Response time: {} seconds'.format(url,response_time))
    except requests.exceptions.Timeout:
        logger.error('Timeout. {} not responding. Response time: {} seconds'.format(url,response_time))
    except requests.exceptions.HTTPError:
        logger.error('HTTP Error. {} returned status code {}. Response time: {} seconds'.format(url,response.status_code, response_time))
    except requests.exceptions.TooManyRedirects:
        logger.error('Too many redirects for {}. Response time: {} seconds'.format(url,response_time))
    except:
        logger.error('Content requirement not found for {}. Response time: {} seconds'.format(url,response_time))

And here where I call this function for all urls:

def async_requests(delay,urls):
    for url in urls:
        async_task = make_async(request,delay,url,10)
        loop.call_soon(delay,async_task)
    try:
        loop.run_forever()
    finally:
        loop.close()

delay argument is interval for loop which describes how often function needs to be executed. In order to loop request I created something like this:

def make_async(func,delay,*args,**kwargs):

    def wrapper(*args, **kwargs):
        func(*args, **kwargs)
        loop.call_soon(delay, wrapper)

    return wrapper

every time I execute async_requests I get this error for each url:

Exception in callback 1.0(<function mak...x7f1d48dd1730>)
handle: <Handle 1.0(<function mak...x7f1d48dd1730>)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
TypeError: 'float' object is not callable

Also request functions for each urls are not being executed periodically as intended. Also my print function which goes after async_requests is not executed either:

async_requests(args.delay,urls)
print("Starting...")

I understand that I'm doing something wrong in code but I can't figure how to solve this problem. I'm beginner in python and not very experienced with asyncio. Summarizing what I want to achive:

Upvotes: 7

Views: 14565

Answers (1)

Mikhail Gerasimov
Mikhail Gerasimov

Reputation: 39606

except:

It'll catch also service exceptions line KeyboardInterrupt or StopIteration. Never do such thing. Instead write:

except Exception:

How to run requests.get asynchronously in Python 3 using asyncio?

requests.get is blocking by nature.

You should either find async alternative for requests like aiohttp module:

async def get(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            return await resp.text()

or run requests.get in separate thread and await this thread asynchronicity using loop.run_in_executor:

executor = ThreadPoolExecutor(2)

async def get(url):
    response = await loop.run_in_executor(executor, requests.get, url)
    return response.text

Upvotes: 14

Related Questions