CODEWITHSUNDEEP
pythonpython-3.xpython-asyncioaiohttp
micheal
micheal

Reputation: 1363

ayncio timeout in wait method explain

>>> import asyncio
>>> help(asyncio.wait)

..

Help on function wait in module asyncio.tasks:

wait(fs, *, loop=None, timeout=None, return_when='ALL_COMPLETED')
    Wait for the Futures and coroutines given by fs to complete.

    Coroutines will be wrapped in Tasks.

    Returns two sets of Future: (done, pending).

    Usage:

        done, pending = yield from asyncio.wait(fs)

    Note: This does not raise TimeoutError! Futures that aren't done
    when the timeout occurs are returned in the second set.
(END)

I dont quite understand last Note in this help (what is second set? is it pending/reprocessing set? how do I execute pending tasks and combine the results of both done and pending and then save in DB)

My problem: I'm using asyncio with aiohttp, have millions of urls , few of them might raise timeout error. I want to send them in a queue for reprocessing or it should take care by eventpool.

import asyncio
import aiohttp
sem = asyncio.Semaphore(10)

def process_data(url):
    with (yield from sem):
          response = yield from aiohttp.request('GET', url)
          print(response)

loop = asyncio.get_event_loop()
c = asyncio.wait([process_data(url) for url in url_list], timeout=10)
loop.run_until_complete(c)

PS: I'm not using wait_for method.

Upvotes: 3

Views: 4419

Answers (1)

leech
leech

Reputation: 8431

Here are the two sets from the help:

Returns two sets of Future: (done, pending).

The second set is the pending set, jobs that haven't finished within the timeout. It will return a tuple with two lists of futures, one of those that are done, and one that are still pending.

instead of:

c = asyncio.wait([process_data(url) for url in url_list], timeout=10)
loop.run_until_complete(c)

you should probably:

def dostuff():
    done, pending = yield from asyncio.wait([process_data(url) for url in url_list], timeout=10)

    # do something with pending

loop.run_until_complete(dostuff())

Here is more information:

https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.wait

Upvotes: 4

Related Questions