CODEWITHSUNDEEP
python-3.xargumentsabstract-syntax-treepython-internals
ninjaconcombre
ninjaconcombre

Reputation: 534

How to modify the signature of a function dynamically

I am writing a framework in Python. When a user declares a function, they do:

def foo(row, fetch=stuff, query=otherStuff)

def bar(row, query=stuff)

def bar2(row)

When the backend sees query= value, it executes the function with the query argument depending on value. This way the function has access to the result of something done by the backend in its scope.

Currently I build my arguments each time by checking whether query, fetch and the other items are None, and launching it with a set of args that exactly matches what the user asked for. Otherwise I got the "got an unexpected keyword argument" error. This is the code in the backend:

#fetch and query is something computed by the backend
if fetch= None and query==None:
    userfunction(row)
elif fetch==None:
    userunction (row, query=query)
elif query == None:
    userfunction (row, fetch=fetch)
else:
    userfunction (row,fetch=fetch,query=query)

This is not good; for each additional "service" the backend offers, I need to write all the combinations with the previous ones.

Instead of that I would like to primarily take the function and manually add a named parameter, before executing it, removing all the unnecessary code that does these checks. Then the user would just use the stuff it really wanted.

I don't want the user to have to modify the function by adding stuff it doesn't want (nor do I want them to specify a kwarg every time).

So I would like an example of this if this is doable, a function addNamedVar(name, function) that adds the variable name to the function function.

I want to do that that way because the users functions are called a lot of times, meaning that it would trigger me to, for example, create a dict of the named var of the function (with inspect) and then using **dict. I would really like to just modify the function once to avoid any kind of overhead.

Upvotes: -2

Views: 514

Answers (2)

Armen Michaeli
Armen Michaeli

Reputation: 9170

You can change the signature of a function by replacing the __code__ property value, e.g. using the convenient replace method:

def f(a, b, c):
    return (a, b, c)

f.__code__ = f.__code__.replace(co_varnames=f.__code__.co_varnames + ('d',), co_nlocals=4, co_argcount=4) # Add the `d` parameter and adjust related property values correspondingly

...but there are major caveats doing so and expecting a generally working function. You'd be surprised to find the function "broken", especially if you're not familiar with Python (CPython) bytecode.

(Everything that follows applies to the CPython implementation of Python, since the behaviour is implementation-dependent.)

Name expressions (e.g. a in return a;) in [compiled] functions address the variable by [a number] and not by its name! This number does not change after compilation and refers to the order of the variable in the list of "locals" (which includes the list of arguments, the way CPython implements it).

When you change the signature, adding or otherwise "reshuffling" elements that are part of the list, the same index ends up pointing to a different variable!

This can be demonstrated if you modify the function by inserting a parameter into the set of function parameters specifically at the front:

def f(a, b, c):
    return (a, b, c)

f.__code__ = f.__code__.replace(co_varnames=('d',) + f.__code__.co_varnames, co_nlocals=4, co_argcount=4) # Observe that `d` is added _before_ the rest of the original set of parameters, in contrast to the expression in the first snippet

Now where with original definition of f the call f('A', 'B', 'C') would return the tuple ('A', 'B', 'C'), the call to modified f, f('D', 'A', 'B', 'C'), will return the tuple... ('D', 'A', 'B')! f accesses arguments by the same index in the ordered list of parameters, that was originally computed when f was compiled!

This is why modifying the signature with __code__ is neither most useful in this scenario nor, well, "safe".

If you use the dis module and the eponymous dis function it exports, you can see the disassembly showing the wrong, "shifted" mapping of indices to names for modified f (as per the last snippet):

>>> dis.dis(f)
  1           0 RESUME                   0

  2           2 LOAD_FAST                0 (d)
              4 LOAD_FAST                1 (a)
              6 LOAD_FAST                2 (b)
              8 BUILD_TUPLE              3
             10 RETURN_VALUE

(the three consecutive LOAD_FAST instructions fetch d, a, and b respectively, although the compiled source expressed "return a, b, and c")

So in conclusion, sure -- you can modify the signature of a function -- but know what to expect.

locals()['a'] will work (and so will inspect) regardless of where a ends up in the [modified] parameter set, probably because the locals built-in has a way of "locating" the variable by name in a manner that doesn't break with changes of parameter set. It's just that the name expressions break, at least.

The same general addressing problem also concerns use of free variables (e.g. those in enclosing closure), by the way.

P.S. Why all the trouble of modifying function arguments in the first place? Well, it's a [problematic] alternative to compiling new function (from source code or AST), which may be used to write decorators, at least.

Upvotes: 1

ninjaconcombre
ninjaconcombre

Reputation: 534

This is indeed doable in AST and that's what I am gonna do because this solution will suit better for my use case . However you could do what I asked more simply by having a function cloning approach like the code snippet I show. Note that this code return the same functions with different defaults values. You can use this code as example to do whatever you want. This works for python3

def copyTransform(f, name, **args):
    signature=inspect.signature(f)
    params= list(signature.parameters)
    numberOfParam= len(params)
    numberOfDefault= len(f.__defaults__)
    listTuple= list(f.__defaults__)

    for key,val in args.items():
        toChangeIndex = params.index(key, numberOfDefault)
        if toChangeIndex:
            listTuple[toChangeIndex- numberOfDefault]=val

    newTuple= tuple(listTuple)
    oldCode=f.__code__

    newCode= types.CodeType(
        oldCode.co_argcount,             #   integer
        oldCode.co_kwonlyargcount,       #   integer
        oldCode.co_nlocals,              #   integer
        oldCode.co_stacksize,            #   integer
        oldCode.co_flags,                #   integer
        oldCode.co_code,                 #   bytes
        oldCode.co_consts,               #   tuple
        oldCode.co_names,                #   tuple
        oldCode.co_varnames,             #   tuple
        oldCode.co_filename,             #   string
        name,                            #   string
        oldCode.co_firstlineno,          #   integer
        oldCode.co_lnotab,               #   bytes
        oldCode.co_freevars,             #   tuple
        oldCode.co_cellvars              #   tuple
        )

    newFunction=types.FunctionType(newCode, f.__globals__, name, newTuple, f.__closure__)
    newFunction.__qualname__=name #also needed for serialization

You need to do that weird stuff with the names if you want to Pickle your clone function.

Upvotes: 0

Related Questions