CODEWITHSUNDEEP
python-asyncio
crusaderky
crusaderky

Reputation: 2692

Handling asyncio deadlocks

This sample code hangs indefinitely:

import asyncio


async def main():
    async def f():
        await g_task

    async def g():
        await f_task

    f_task = asyncio.create_task(f())
    g_task = asyncio.create_task(g())
    await f_task


asyncio.run(main())

I'm looking for a way to automatically detect and handle deadlocks, like GoLang does.

So far I came up with a variant of asyncio.wait_for():

[EDIT] overhauled design

https://gist.github.com/gimperiale/549cbad04c24d870145d3f38fbb8e6f0

1 line change in the original code:

await wait_check_deadlock(f_task)

It works, but with two major problems:

  1. it relies on asyncio.Task._fut_waiter, which is an implementation detail of CPython
  2. The deadlocked tasks will remain in RAM forever. aw.cancel() seems to do nothing. If I catch the RecursionError my helper function raises, asyncio.run() raises another RecursionError when it tries cancelling all tasks.

Are there more robust solutions to the problem?

Upvotes: 7

Views: 3646

Answers (1)

Dima Tisnek
Dima Tisnek

Reputation: 11781

Deadlock avoidance has been researched a lot, some practical solutions exist, but in general case, the problem is undecidable (I think it can be reduced to the halting problem).

To illustrate practicality, consider this:

await asyncio.sleep(2 ** (1 / random.random()))

Depending on your luck, it will either return soon or "practically never".

This trick can be used to show that callback-based program is impossible to predict:

f = asyncio.Future()

async foo():
    await asyncio.sleep(2 ** (1 / random.random()))
    f.set_result(None)

async bar():
    await f

await asyncio.gather(foo(), bar())

Likewise, it can be applied to your "pure" async/await program:

async def f():
    await g_task

async def g():
    await asyncio.wait(f_task,
                       asyncio.sleep(2 ** (1 / random.random())),
                       return_when=asyncio.FIRST_COMPLETED)

f_task = asyncio.create_task(f())
g_task = asyncio.create_task(g())
await f_task

At the same time, imperfect but practical deadlock detector can be very useful, please consider posting your code to core asyncio devs and/or a standalone library.

The current practice is to run tests with PYTHONASYNCIODEBUG=1 which shows unawaited tasks (destroyed before result / exception was read).

Your library could be better, for example, it could report when some task took longer than X, or when a DAG of tasks depending on given task grows too large.

Upvotes: 3

Related Questions