True dynamic and anonymous functions possible in Python?

Question

Just as a dynamic class can be created using type(name, base-classes, namespace-dict), can a dynamic function be created?

I've tried doing something along the lines of:

>>> f = type("f", (function,), {})
NameError: name 'function' is not defined

Ok, so I'll be clever, but:

>>> def fn():
...   pass
... 
>>> type(fn)
<type 'function'>
>>> f = type("f", (type(fn),), {})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'function' is not an acceptable base type

Does Python specifically prevent the creation of dynamic functions in the same way it allows dynamic classes?

Edit: Note, I'd disallow any use of exec.. Since my question is does the Python language itself permit this.

Thanks in advance.

Answer 1

python provides limited support for anonymous executable objects, and that is lambda, though it does not support statements, only expressions.

This is due to the syntax of python, and there will be no simple workarounds.

If you need to generate more complex executable objects, then don't be anonymous, just create a function object in local scope with a non-conflicting name.

These are described in the documentation.

Python lambda expressions cannot contain statements because Python’s syntactic framework can’t handle statements nested inside expressions. However, in Python, this is not a serious problem. Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you’re too lazy to define a function.

Functions are already first class objects in Python, and can be declared in a local scope. Therefore the only advantage of using a lambda instead of a locally-defined function is that you don’t need to invent a name for the function – but that’s just a local variable to which the function object (which is exactly the same type of object that a lambda expression yields) is assigned!

https://docs.python.org/3.7/faq/design.html#why-can-t-lambda-expressions-contain-statements

Answer 2

You can avoid generating then execing source code if you're ready to generate Abstract Syntax Trees (AST's) and compile them instead. It might be slightly better because data can stay structured all along.

from ast import *
from types import *

function_ast = FunctionDef(
    name='f',
    args=arguments(args=[], vararg=None, kwarg=None, defaults=[]),
    body=[Return(value=Num(n=42, lineno=1, col_offset=0), lineno=1, col_offset=0)],
    decorator_list=[],
    lineno=1,
    col_offset=0
)
module_ast = Module(body=[function_ast])

module_code = compile(module_ast, "<not_a_file>", "exec")
function_code = [c for c in module_code.co_consts if isinstance(c, CodeType)][0]

f = FunctionType(function_code, {})

print f()

The code above will print 42.

To get inspiration about what the generated AST should be, you can use:

print(dump(parse("def f(): return 42"), include_attributes=True))

Of course, ASTs are different in Python 2 and Python 3.

Edit:

Tested and working in Python 3.8

from ast import *
from types import *

function_ast = FunctionDef(
    name='f',
    args=arguments(
        args=[], vararg=None, kwarg=None, defaults=[],
        kwonlyargs=[], kw_defaults=[], posonlyargs=[]
    ),
    body=[Return(value=Num(n=42, lineno=1, col_offset=0), lineno=1, col_offset=0)],
    decorator_list=[],
    lineno=1,
    col_offset=0
)
module_ast = Module(body=[function_ast], type_ignores=[])

module_code = compile(module_ast, "<not_a_file>", "exec")
function_code = [c for c in module_code.co_consts if isinstance(c, CodeType)][0]

f = FunctionType(function_code, {})

print(f())

Answer 3

jacquev6's solution works quite well for me after updating it for Python 3 by adding kwonlyargs=[], kw_defaults=[] to the call to arguments() :

#!/usr/bin/env python3

from ast import *
from types import *

function_ast = FunctionDef(
    name='f',
    args=arguments(args=[], vararg=None, kwarg=None, defaults=[], kwonlyargs=[], kw_defaults=[]),
    body=[Return(value=Num(n=42, lineno=1, col_offset=0), lineno=1, col_offset=0)],
    decorator_list=[],
    lineno=1,
    col_offset=0
)
module_ast = Module(body=[function_ast])
module_code = compile(module_ast, "<not_a_file>", "exec")
function_code = [c for c in module_code.co_consts if isinstance(c, CodeType)][0]

f = FunctionType(function_code, {})

print(f())

Answer 4

There is types.FunctionType which you can use to dynamically create a function e.g.

def test_func(): print 'wow' 
dynf = types.FunctionType(test_func.func_code, {})
dynf()

Output:

wow

You might object that this is not dynamic because I am using code from another function, but that was just an example there is a way to generate code from python strings e.g.

dynf = types.FunctionType(compile('print "really WoW"', 'dyn.py', 'exec'), {})
dynf()

Output:

really WoW

Now that is dynamic!

OP is worried about the dynamic nature of such function so here is another example

dynf = types.FunctionType(compile('test_func():\ntest_func()', 'dyn.py', 'exec'), globals())
dynf()

Output:

wow
wow

Note: Creating Function object like this seems to have limitations e.g. it is not easy to pass arguments, because to pass arguments we need to pass correct co_argcount, co_varnames and other 12 variables to types.CodeType, which theoretically can be done but will be error prone, an easier way is to import string as a module and you have a full fledged function e.g.

import types
import sys,imp

code = """def f(a,b,c):
    print a+b+c, "really WoW"
"""
module = imp.new_module('myfunctions')
exec code in module.__dict__
module.f('W', 'o', 'W')

Output:

WoW really WoW

Answer 5

Yes. Use the def statement:

def functor(f): # this dynamically creates a function at module load time
    def inner(g): #this dynamically creates a function on execution of functor
        return f(g)

    return inner

Any solution involving freestanding text being compiled is equivalent to exec or eval, which leaves you with pre-existing functions and lexically captured data items to stitch together new functions. This might give you some ideas.

Answer 6

A lot of people seem to be misled about the purpose of “lambda” in Python: its sole purpose is to define a simple single-expression function without a name. Nothing more. In Python, functions are indeed first-class objects, like they are in, say, LISP: you can pass them as arguments, store them in data structures, and return them as results. For example, here is a function that composes two given functions f and g, so compose(f, g)(x) is equivalent to f(g(x)):

def compose(f, g) :
    def result(x) :
        return f(g(x))
    #end result
    return result
#end compose

and here’s an example use:

>>> h = compose(lambda x : x + 2, lambda x : x * x)
>>> h(3)
11

Answer 7

Python does allow creation of dynamic functions. One approach is using lambda:

>>> g = lambda x: x**2
>>> g
<function <lambda> at 0xa68c924>
>>> g(3)
9
>>> g = lambda x: x*2
>>> g
<function <lambda> at 0xa68c95c>
>>> g(3)
6
>>> 

Another approach is described here: Lexical closures in Python

So, you don't need the hocus-pocus of the behavioral patterns like Strategy.

It would be useful if you could tell us the problem you want to solve so we could find could out which language constructs are appropriate for that.

Answer 8

You'll want to look into collections.Callable, which is just a good place to start when defining __call__.

from collections import Callable
class SomeCallableClass(Callable):
    def __call__(self, x):
        print(x)

some_function = SomeCallableClass()
some_function(1)

Will give us 1 as out output. This allows you to construct functions at will.

from collections import Callable
class SomeCallableClass(Callable):
    def __init__(self, n):
        self.n = n
    def __call__(self, x):
        for i in range(self.n):
            print(x)

some_function = SomeCallableClass(2)
some_function("Two times.")
some_function = SomeCallableClass(3)
some_function("Three times.")

Which gives us:

Two times.
Two times.
Three times.
Three times.
Three times.

You can use this to construct functions as complex as you want.