CODEWITHSUNDEEP
pythonpython-asyncio
0x5453
0x5453

Reputation: 13589

Python asyncio: Enter into a temporary async context?

I want to write a library that mixes synchronous and asynchronous work, like:

def do_things():
    # 1) do sync things
    # 2) launch a bunch of slow async tasks and block until they are all complete or an exception is thrown
    # 3) more sync work
    # ...

I started implementing this using asyncio as an excuse to learn the learn the library, but as I learn more it seems like this may be the wrong approach. My problem is that there doesn't seem to be a clean way to do 2, because it depends on the context of the caller. For example:

Basically it seems like if I want to use asyncio at all, I am forced to use it for the entire lifetime of the program, and therefore all libraries like this one have to be written as either 100% synchronous or 100% asynchronous. But the behavior I want is: Use the current event loop if one is running, otherwise create a temporary one just for the scope of 2, and don't break client code in doing so. Does something like this exist, or is asyncio the wrong choice?

Upvotes: 1

Views: 1055

Answers (1)

user4815162342
user4815162342

Reputation: 155046

I can't use asyncio.run(), because the caller could already have a running event loop and you can only have one loop per thread.

If the caller has a running event loop, you shouldn't run blocking code in the first place because it will block the caller's loop!

With that in mind, your best option is to indeed make do_things async and call sync code using run_in_executor which is designed precisely for that use case:

async def do_things():
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, sync_stuff)
    await async_func()
    await loop.run_in_executor(None, more_sync_stuff)

This version of do_things is usable from async code as await do_things() and from sync code as asyncio.run(do_things()).

Having said that... if you know that the sync code will run very briefly, or you are for some reason willing to block the caller's event loop, you can work around the limitation by starting an event loop in a separate thread:

def run_async(aw):
    result = None
    async def run_and_store_result():
        nonlocal result
        result = await aw
    t = threading.Thread(target=asyncio.run, args=(run_and_store_result(),))
    t.start()
    t.join()
    return result

do_things can then look like this:

async def do_things():
    sync_stuff()
    run_async(async_func())
    more_sync_stuff()

It will be callable from both sync and async code, but the cost will be that:

  • it will create a brand new event loop each and every time. (Though you can cache the event loop and never exit it.)
  • when called from async code, it will block the caller's event loop, thus effectively breaking its asyncio usage, even if most time is actually spent inside its own async code.

Upvotes: 3

Related Questions