CODEWITHSUNDEEP
pythonpython-3.xpython-asyncio
Dustin Oprea
Dustin Oprea

Reputation: 10236

Proper way of managing recursive Python async tasks

My understanding of async tasks is that they can be created and the returned task object itself can just be discarded because the task will automatically go on the loop, and then asyncio.all_tasks() can be called later to join them. However, hitting all_tasks() once won't account for any tasks created from tasks, and, as a result, an exception will get raised but not propagated. Even after the task that dispatches its own task is executed, asyncio.all_tasks() still doesn't seem to see it.

So, do we actually need proper task accounting and to make sure to gather/run on all created tasks? It seems like this is partially managed, but the important stuff is not.

Example code:

import asyncio

async def func2():
    print("Second function")

    raise Exception("Inner exception")

async def func1(loop):
    print("First function")

    loop.create_task(func2())

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

awaitable = func1(loop)
loop.create_task(awaitable)

reported_tasks = asyncio.all_tasks(loop=loop)

awaitable = asyncio.gather(*reported_tasks)
loop.run_until_complete(awaitable)

I'm creating the loop myself because my scenario is a non-main thread (so the loop won't be implicitly created) and I don't want there to be some nuance due to this that confuses things due to being left unspoken.

func1() and func2() execute, but I'll merely just get warned about not materializing the exception in func2() unless I manually collect all of the tasks, enumerate them, and check their exception flags.

On the other hand, if I start adding all of the tags to a list, do I grab that list, get the length, shift that number of items off the front of the list, wait on those to be done, check them for exceptions, and loop until I determine that all tasks have completed? This will potentially include some long-running tasks as well as some failed ones, so maybe I need to figure for a timeout in order to make sure those exceptions get processed seen after being raised? Is there a more elegant solution?

I looked, but the documentation seems mostly about the API rather than the use-cases/patterns and all of the searches I ran still left these questions unanswered.

The documentation does note that you should retain the created tasks and implies that this is for the reason of reference-counting, but there's no reference to the topics above:

Important
Save a reference to the result of this function, to avoid a task disappearing mid execution.

Upvotes: 0

Views: 594

Answers (1)

jsbueno
jsbueno

Reputation: 110271

The problem with your snippet seems to be only that you call all_tasks before the body of func1 is run, and therefore, before the task that will run func2 is ever created.

I typed some stuff on the interactive interpreter, without all the boilerplate you added there, and all_tasks seems to be "seeing" the grand-daughter task, with no problems, and the exception is also raised in the awaiting code:


In [1]: import asyncio

In [2]: async def func2():
   ...:     print("second function")
   ...:     1 / 0  # raises
   ...: 

In [3]: async def func1():
   ...:     print("first function")
   ...:     asyncio.create_task(func2())
   ...: 

In [4]: async def main():
   ...:     await func1()
   ...:     z = asyncio.all_tasks()
   ...:     print(z)
   ...: 

In [5]: asyncio.run(main())
first function
{<Task pending name='Task-1148' coro=<main() running at <ipython-input-4-489680e1b4c1>:4> cb=[_run_until_complete_cb() at /home/gwidion/.pyenv/versions/3.10-dev/lib/python3.10/asyncio/base_events.py:184]>, <Task pending name='Task-1149' coro=<func2() running at <ipython-input-2-555de5f64f41>:1>>}
second function
Task exception was never retrieved
future: <Task finished name='Task-1149' coro=<func2() done, defined at <ipython-input-2-555de5f64f41>:1> exception=ZeroDivisionError('division by zero')>
Traceback (most recent call last):
  File "<ipython-input-2-555de5f64f41>", line 3, in func2
    1 / 0  # raises
ZeroDivisionError: division by zero

Upvotes: 1

Related Questions