async exec in python

Question

I'd like to call exec in an async function and do something like the following code (which is not valid):

import asyncio

async def f():
    await exec('x = 1\n' 'await asyncio.sleep(x)')

More precisely, I'd like to be able to wait for a future inside the code that runs in exec.

How can this be achieved?

Answer 1

Based on the different answers, the only thing that I missed was the local variables (couldn't make them be global).

Here is what I did :

async def async_eval_or_exec(message, vars):
    # Translating the command into a function
    # The function returns the main result AND the local variables to make them accessible outside

    if message.count("\n"):
        function = "async def __execute__():\n    " + message.replace("\n", "    ") + "\n    return None, locals()\n"
    else:
        function = "async def __execute__():\n    return " + message + ", locals()\n"

    # The execution - get the result of 
    try:
        exec(function, vars, vars)
        result = await vars["__execute__"] ()
        if result[ 0 ] is not None:
            return result[ 0 ]
        vars.update(result[ 1 ]) # forces the local variables (inside __execute__) to be global
    except SyntaxError: # for commands like "import os"
        exec(message, vars, vars)

And then, I can run :

>>> vars = {}
>>> await async_eval_or_exec('x = 1', vars)
>>> await async_eval_or_exec('await asyncio.sleep(x)', vars)
>>> await async_eval_or_exec('print(x)', vars)
1
>>> 

It can be useful if you create an async interpreter (to not lose objects you create inside the execute function).

I think it is better than using the "global" command every time you create a variable.

Answer 2

As of Python 3.8, you can compile() the code with the flag ast.PyCF_ALLOW_TOP_LEVEL_AWAIT and eval it to get a coroutine back, which you can then await. Despite using eval, multiple statements are supported.

Here is a minimal example of how to do this:

import ast
import asyncio

async def main() -> None:
    code = compile(
        "x = 1\n"
        "await asyncio.sleep(0.1)\n"
        "print('done!')\n",
        "<string>",
        "exec",
        flags=ast.PyCF_ALLOW_TOP_LEVEL_AWAIT,
    )
    coroutine: Awaitable | None = eval(code)
    if coroutine is not None:
        await coroutine

asyncio.run(main())

When there are no await statements, eval() runs the code immediately then returns None.

Answer 3

Just use this function:

import asyncio
async def async_exec(code):
    t = [None]
    exec('async def _async_exec():\n return {}\nt[0] = asyncio.ensure_future(_async_exec())'.format(code))
    return await t[0]

Here is a code example that can directly run. (It works for Python 3.6.8)

import asyncio
async def async_exec(code):
    t = [None]
    exec('async def _async_exec():\n return {}\nt[0] = asyncio.ensure_future(_async_exec())'.format(code))
    return await t[0]

async def p(s):
    await asyncio.sleep(s)
    return s


async def main():
    print(await async_exec('await p(0.1) / await p(0.2)'))


asyncio.get_event_loop().run_until_complete(main())

I try to explain it, define an async function in exec. Inside the async function, run the code you want. But exec doesn't have return value, use t[0] to store an asyncio future, await the future outside of exec to get the return value.

Answer 4

Here is a more robust way using the builtin ast module:

import ast

async def async_exec(stmts, env=None):
    parsed_stmts = ast.parse(stmts)

    fn_name = "_async_exec_f"

    fn = f"async def {fn_name}(): pass"
    parsed_fn = ast.parse(fn)

    for node in parsed_stmts.body:
        ast.increment_lineno(node)

    parsed_fn.body[0].body = parsed_stmts.body
    exec(compile(parsed_fn, filename="<ast>", mode="exec"), env)

    return await eval(f"{fn_name}()", env)

Answer 5

Here's a module using AST to do stuff. This means that mutli-line strings will work perfectly and line-numbers will match the original statements. Also, if anything is an expression, it is returned (as a list if there are multiple, otherwise as just an element)

I made this module (check the revision history of this answer for more details on the inner workings). I use it here

Answer 6

This is based off @YouTwitFace's answer, but keeps globals unchanged, handles locals better and passes kwargs. Note multi-line strings still won't keep their formatting. Perhaps you want this?

async def aexec(code, **kwargs):
    # Don't clutter locals
    locs = {}
    # Restore globals later
    globs = globals().copy()
    args = ", ".join(list(kwargs.keys()))
    exec(f"async def func({args}):\n    " + code.replace("\n", "\n    "), {}, locs)
    # Don't expect it to return from the coro.
    result = await locs["func"](**kwargs)
    try:
        globals().clear()
        # Inconsistent state
    finally:
        globals().update(**globs)
    return result

It starts by saving the locals. It declares the function, but with a restricted local namespace so it doesn't touch the stuff declared in the aexec helper. The function is named func and we access the locs dict, containing the result of the exec's locals. The locs["func"] is what we want to execute, so we call it with **kwargs from aexec invocation, which moves these args into the local namespace. Then we await this and store it as result. Finally, we restore locals and return the result.

Warning:

Do not use this if there is any multi-threaded code touching global variables. Go for @YouTwitFace's answer which is simpler and thread-safe, or remove the globals save/restore code

Answer 7

Note: F-strings are only supported in python 3.6+. For older versions, use %s, .format() or the classic + concatenation.

async def aexec(code):
    # Make an async function with the code and `exec` it
    exec(
        f'async def __ex(): ' +
        ''.join(f'\n {l}' for l in code.split('\n'))
    )

    # Get `__ex` from local variables, call it and return the result
    return await locals()['__ex']()

---

Known issues:

• If you use new lines in a string (triple quotes), it will mess up the formatting.

Answer 8

Thanks for all the suggestions. I figured out that this can be done with greenlets along async, since greenlets allow performing "top level await":

import greenlet
import asyncio

class GreenAwait:
    def __init__(self, child):
        self.current = greenlet.getcurrent()
        self.value = None
        self.child = child

    def __call__(self, future):
        self.value = future
        self.current.switch()

    def __iter__(self):
        while self.value is not None:
            yield self.value
            self.value = None
            self.child.switch()

def gexec(code):
    child = greenlet.greenlet(exec)
    gawait = GreenAwait(child)
    child.switch(code, {'gawait': gawait})
    yield from gawait

async def aexec(code):
    green = greenlet.greenlet(gexec)
    gen = green.switch(code)
    for future in gen:
        await future

# modified asyncio example from Python docs
CODE = ('import asyncio\n'
        'import datetime\n'

        'async def display_date():\n'
        '    for i in range(5):\n'
        '        print(datetime.datetime.now())\n'
        '        await asyncio.sleep(1)\n')

def loop():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(aexec(CODE + 'gawait(display_date())'))
    loop.close()

Answer 9

Yours problem is that you are trying to await to None object- exec ignores the return value from its code, and always returns None. If you want to execute and await to the result you should use eval- eval returns the value of the given expression.

Your's code should look like this:

import asyncio

async def f():
    exec('x = 1')
    await eval('asyncio.sleep(x)')

loop = asyncio.get_event_loop()
loop.run_until_complete(f())
loop.close()