CODEWITHSUNDEEP
pythonweb-scrapingpython-asynciopython-requests-html
Marco Antonio
Marco Antonio

Reputation: 143

Creating multiple async requests

I'm just starting to explore this new library called Requests-Html, which I've just dicovered through Corey Schafer tutorial, and the challenge is to create a async call of n different requests.

So, for example, we have the following code, which runs in about 3,6 seconds:

async def get_delay1():
    r = await asession.get("https://httpbin.org/delay/1")
    return r

async def get_delay2():
    r = await asession.get("https://httpbin.org/delay/2")
    return r

async def get_delay3():
    r = await asession.get("https://httpbin.org/delay/3")
    return r

asession = AsyncHTMLSession()

t1 = time.perf_counter()

results = asession.run(get_delay1, get_delay2, get_delay3)

for result in results:
    response = result.html.url
    print(response)

t2 = time.perf_counter()

print(t2 - t1)

The question is, what should I do if I want to create a 500 async request with this library? It can't be the case that I will have to code 500 different functions, right?

I've tried to create a list with the functions generators, so that I can automatically pass n different functions inside of it, :

tasks = [get_delay1, get_delay2, get_delay3]
results = asession.run(tasks)

But I get the 

ERROR`: 
asyncio.ensure_future(coro()) for coro in coros
TypeError: 'list' object is not callable

Upvotes: 0

Views: 552

Answers (1)

Marco Antonio
Marco Antonio

Reputation: 143

I've found out how to do it.

After we create our list, we have to call

asession.run(*tasks)

This way, we ensure that we are passing only callables to the run method, i.e., elements of the list and not the list itself.

By!

Upvotes: 1

Related Questions