CODEWITHSUNDEEP
python-3.xcode-generation
budchan chao
budchan chao

Reputation: 327

Runtime compiling a function with arguments in python

I am trying to use compile to runtime generate a Python function accepting arguments as follows.

import types
import ast

code = compile("def add(a, b): return a + b", '<string>', 'exec')
fn = types.FunctionType(code, {}, name="add")
print(fn(4, 2))

But it fails with

TypeError: <module>() takes 0 positional arguments but 2 were given

Is there anyway to compile a function accepting arguments using this way or is there any other way to do that?

Upvotes: 1

Views: 1721

Answers (3)

Jay Obermark
Jay Obermark

Reputation: 184

I think this accomplishes what you want in a better way

import types

text = "lambda (a, b): return a + b"
code = compile(text, '<string>', 'eval')
body = types.FunctionType(code, {})
fn = body()

print(fn(4, 2))

The function being anonymous resolves the implicit namespace issues. And returning it as a value by using the mode 'eval' is cleaner that lifting it out of the code contents, since it does not rely upon the specific habits of the compiler.

More usefully, as you seem to have noticed but not gotten to using yet, since you import ast, the text passsed to compile can actually be an ast object, so you can use ast transformation on it.

import types
import ast
from somewhere import TransformTree

text = "lambda (a, b): return a + b"
tree = ast.parse(text)
tree = TransformTree().visit(tree)
code = compile(text, '<string>', 'eval')
body = types.FunctionType(code, {})
fn = body()

print(fn(4, 2))

Upvotes: 0

Edward Minnix
Edward Minnix

Reputation: 2947

Compile returns the code object to create a module. In Python 3.6, if you were to disassemble your code object:

>>> import dis
>>> dis.dis(fn)
 0 LOAD_CONST    0 (<code object add at ...., file "<string>" ...>)
 2 LOAD_CONST    1 ('add')
 4 MAKE_FUNCTION 0
 6 STORE_NAME    0 (add)
 8 LOAD_CONST    2 (None)
10 RETURN_VALUE

That literally translates to make function; name it 'add'; return None.

This code means that your function runs the creation of the module, not returning a module or function itself. So essentially, what you're actually doing is equivalent to the following:

def f():
    def add(a, b):
        return a + b

print(f(4, 2))

For the question of how do you work around, the answer is it depends on what you want to do. For instance, if you want to compile a function using compile, the simple answer is you won't be able to without doing something similar to the following.

# 'code' is the result of the call to compile.
# In this case we know it is the first constant (from dis),
# so we will go and extract it's value
f_code = code.co_consts[0]
add = FunctionType(f_code, {}, "add")

>>> add(4, 2)
6

Since defining a function in Python requires running Python code (there is no static compilation by default other than compiling to bytecode), you can pass in custom globals and locals dictionaries, and then extract the values from those.

glob, loc = {}, {}
exec(code, glob, loc)

>>> loc['add'](4, 2)
6

But the real answer is if you want to do this, the simplest way is generally to generate Abstract Syntax Trees using the ast module, and compiling that into module code and evaluating or executing the module.

If you want to do bytecode transformation, I'd suggest looking at the codetransformer package on PyPi.

TL;DR using compile will only ever return code for a module, and most serious code generation is done either with ASTs or by manipulating byte codes.

Upvotes: 4

smarie
smarie

Reputation: 5156

is there any other way to do that?

For what's worth: I recently created a @compile_fun goodie that considerably eases the process of applying compile on a function. It relies on compile so nothing different than was explained by the above answers, but it provides an easier way to do it. Your example writes:

@compile_fun
def add(a, b):
    return a + b

assert add(1, 2) == 3

You can see that you now can't debug into add with your IDE. Note that this does not improve runtime performance, nor protects your code from reverse-engineering, but it might be convenient if you do not want your users to see the internals of your function when they debug. Note that the obvious drawback is that they will not be able to help you debug your lib, so use with care!

See makefundocumentation for details.

Upvotes: 0

Related Questions