CODEWITHSUNDEEP
pythonperformancepython-3.xbenchmarkingpython-asyncio
okoboko
okoboko

Reputation: 4492

Asyncio Making HTTP Requests Slower?

I'm using Asyncio and Requests to benchmark a series of HTTP requests.

For some reason, it's slightly slower to use Asyncio than just straight Requests. Any idea why? Am I using Asyncio incorrectly?

import asyncio
import functools
import requests
import time

ts = time.time()
for i in range(10):
  @asyncio.coroutine
  def do_checks():
      loop = asyncio.get_event_loop()
      req = loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3))
      resp = yield from req
      print(resp.status_code)

  loop = asyncio.get_event_loop()
  loop.run_until_complete(do_checks())
te = time.time()
print("Version A: " + str(te - ts))

ts = time.time()
for i in range(10):
  r = requests.get("http://google.com", timeout=3)
  print(r.status_code)
te = time.time()

print("Version B:  " + str(te - ts))

Output:

Version A = Asyncio; Version B = Requests

200
200
200
200
200
200
200
200
200
200
Version A: 5.7215821743011475
200
200
200
200
200
200
200
200
200
200
Version B:  5.320340156555176

Upvotes: 7

Views: 4058

Answers (3)

brunsgaard
brunsgaard

Reputation: 5168

Just for completeness, here is a really fast asyncio implementation

import aiohttp
import asyncio
import time

async def main(n):
    ts = time.time()
    session = aiohttp.ClientSession()
    fs = [session.get('http://google.com') for _ in range(n)]
    for f in asyncio.as_completed(fs):
        resp = await f
        print(resp.status)
        await resp.release()
    await session.close()
    te = time.time()
    print("Aiohttp version:  " + str(te - ts))

loop = asyncio.get_event_loop()
loop.run_until_complete(main(10))
loop.close()

The code is python 3.5 and higher.

~> python asyncioreq.py
200
...
Aiohttp version:  0.15974688529968262

Hope somebody can use it ;)

Upvotes: 7

alairock
alairock

Reputation: 1942

Building on @brunsgaard's answer: you can go a step further with aiohttp and gathering your results with asyncio.gather(). You can then take the responses from your requests and handle them.

import aiohttp
import asyncio
import time

async def main(n):
    start = time.time()
    session = aiohttp.ClientSession()
    jobs = [session.get('http://google.com') for _ in range(n)]
    done_jobs = await asyncio.gather(*jobs)
    for done_job in done_jobs:
        print(done_job.status)
    session.close()
    end = time.time()
    print("Time:  " + str(end - start))

loop = asyncio.get_event_loop()
loop.run_until_complete(main(10))
loop.close()

Upvotes: 4

leech
leech

Reputation: 8431

You are waiting for each request to finish before you start the next one. So you have the overhead of the event loop with no benefits.

Try this:

import asyncio
import functools
import requests
import time

ts = time.time()
loop = asyncio.get_event_loop()

@asyncio.coroutine
def do_checks():
    futures = []
    for i in range(10):
        futures.append(loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3)))

    for req in asyncio.as_completed(futures):
        resp = yield from req
        print(resp.status_code)

loop.run_until_complete(do_checks())
te = time.time()
print("Version A: " + str(te - ts))

ts = time.time()
for i in range(10):
    r = requests.get("http://google.com", timeout=3)
    print(r.status_code)
te = time.time()
print("Version B:  " + str(te - ts))

This is what I get when i run it:

$ python test.py 
200
...
Version A: 0.43438172340393066
200
...
Version B: 1.6541109085083008

Much faster, but really this is just spawning threads and waiting for the http library to finish, you don't need asyncio to do that.

You might want to checkout aiohttp as it was built for use with asyncio. requests is a fabulous library, but it is not made for asyncio.

Upvotes: 14

Related Questions