CODEWITHSUNDEEP
pythonweb-crawlerpython-asynciocoroutinehttpx
LudwigWS
LudwigWS

Reputation: 23

When using Python asyncio corurrent send network request, how to make coroutine prefer continue to handle response first, not send a new request?

If I want to request a API 1000 times (send network request and handle response), it will begin to handle response after sending all 1000 request first, and then handle response.

Can I tell asyncio prefer to return back await position code if it's done?

import asyncio

import httpx

# request_time = 10
request_time = 1000 # large enough to ensure previous responses return

limits = httpx.Limits(max_connections=5)
client = httpx.AsyncClient(limits=limits)


async def request_baidu(i):
    # async with httpx.AsyncClient(limits=limits) as client:
        print(f"===> %d" % i)
        r = await client.get("http://www.baidu.com")
        # print(r.status_code)
        print(f"<=== %d" % i) # How to ensure return to run this code, not make a new request (run a new task `request_baidu` here)


async def main():
    request_list = [asyncio.create_task(request_baidu(i)) for i in range(request_time)]
    await asyncio.gather(*request_list)

if __name__ == '__main__':
    asyncio.run(main())

Result

# request_time = 10

===> 0
===> 1
===> 2
===> 3
===> 4
===> 5
===> 6
===> 7
===> 8
===> 9
<=== 3 # (we can see it continue to handle response after sending all request)
<=== 4
<=== 0 
<=== 1
<=== 2
<=== 5
<=== 6
<=== 7
<=== 8
<=== 9

Expect result：

===> 0
===> 1
<=== 0 #(continue handle response when there is some response)
===> 2
===> 3
<=== 1
<=== 2
===> 4
# ...

Upvotes: 2

Views: 633

Answers (1)

HTF
HTF

Reputation: 7290

You can use asyncio.as_completed to get the earliest next result:

test.py:

import asyncio
import time

from httpx import AsyncClient, Limits

REQUESTS = 10
URL = "http://www.baidu.com"
TIMEOUT = 5


async def request_baidu(client, i):
    t1 = time.process_time()
    print(f"===> {i}")
    r = await client.get(f"{URL}", timeout=TIMEOUT)

    return f"<=== {i} ({time.process_time() - t1:.3f}s)"


async def main():
    async with AsyncClient() as client:
        aws = [asyncio.create_task(request_baidu(client, i)) for i in range(REQUESTS)]

        for coro in asyncio.as_completed(aws):
            earliest_result = await coro
            print(earliest_result)


if __name__ == "__main__":
    asyncio.run(main())

Test:

$ python test.py
===> 0
===> 1
===> 2
===> 3
===> 4
===> 5
===> 6
===> 7
===> 8
===> 9
<=== 9 (0.073s)
<=== 2 (0.081s)
<=== 1 (0.082s)
<=== 0 (0.086s)
<=== 8 (0.086s)
<=== 4 (0.088s)
<=== 6 (0.090s)
<=== 3 (0.092s)
<=== 7 (0.092s)
<=== 5 (0.093s)

Upvotes: 3

Related Questions