CODEWITHSUNDEEP
pythonasynchronoustaskfuture
Blark
Blark

Reputation: 307

Re-queue futures after exception with asyncio

I'm trying to figure out how to re-queue some asynchronous DNS requests that have timed out (I'm using uvloop and aiodns modules).

Here's the code that I set up the loop with:

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
sem = asyncio.Semaphore(MAX_COUNT)

This function performs the lookup: 

async def lookup(name):
    with (await sem):
        response = await resolver.query(name, 'A')
        return response

I read in a file containing names to look up and set up tasks including a callback to deal with the results:

for n in names:
    host = '{}.{}'.format(n, domain)
    task = asyncio.ensure_future(lookup(host))
    tasks.append(task)
    task.add_done_callback(functools.partial(got_result, host))

And start the lookup queue. 

print("Looking up {} subdomains...".format(len(names)))
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

In the got_result callback I then test for future.exception() and deal with it if there is one, and if not print the result to screen. Some of the exceptions I'm fine with (i.e. domain name not found), but others like a timeout I want to re-queue the item. Is there an easy-way to add the future back to the loop for another try or do I need to set up a separate function for that and re-add a task manually?

Thanks for your help.

Upvotes: 1

Views: 1956

Answers (1)

stovfl
stovfl

Reputation: 15513

Question: ...but others like a timeout I want to re-queue the item.

Requeue the Task could lead to a Deadlock.
Instead of requeue the task, hold the Task, for instance:

async def lookup(name):
    with (await sem):
        retries = 0
        while retries <= 5:
            try:
                response = await resolver.query(name, 'A')
                break
            except TimeoutError:
                retries += 1
                yield from asyncio.sleep(1)

        return response

Upvotes: 5

Related Questions