Reputation: 27160

Preserving signatures of decorated functions

Suppose I have written a decorator that does something very generic. For example, it might convert all arguments to a specific type, perform logging, implement memoization, etc.

Here is an example:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

Everything well so far. There is one problem, however. The decorated function does not retain the documentation of the original function:

>>> help(funny_function)
Help on function g in module __main__:

g(*args, **kwargs)

Fortunately, there is a workaround:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

This time, the function name and documentation are correct:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

But there is still a problem: the function signature is wrong. The information "*args, **kwargs" is next to useless.

What to do? I can think of two simple but flawed workarounds:

1 -- Include the correct signature in the docstring:

def funny_function(x, y, z=3):
    """funny_function(x, y, z=3) -- computes x*y + 2*z"""
    return x*y + 2*z

This is bad because of the duplication. The signature will still not be shown properly in automatically generated documentation. It's easy to update the function and forget about changing the docstring, or to make a typo. [And yes, I'm aware of the fact that the docstring already duplicates the function body. Please ignore this; funny_function is just a random example.]

2 -- Not use a decorator, or use a special-purpose decorator for every specific signature:

def funny_functions_decorator(f):
    def g(x, y, z=3):
        return f(int(x), int(y), z=int(z))
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

This works fine for a set of functions that have identical signature, but it's useless in general. As I said in the beginning, I want to be able to use decorators entirely generically.

I'm looking for a solution that is fully general, and automatic.

So the question is: is there a way to edit the decorated function signature after it has been created?

Otherwise, can I write a decorator that extracts the function signature and uses that information instead of "*kwargs, **kwargs" when constructing the decorated function? How do I extract that information? How should I construct the decorated function -- with exec?

Any other approaches?

Upvotes: 147

Answers (9)

Daraan

Reputation: 3780

As already answered functools.wraps is your way to go. This sometimes has downside that in IDE type-checkers/linters that use typeshed the type-hint in your IDE of even a simple function changes, changes in a not so nicely readable _Wrapped type. You need to cast or overwrite that type hint again, e.g. with the signature of a decorator that that is used alongside.

@wraps(int)
def foo(x: str):
    return int(x)

reveal_type(foo)
# foo: _Wrapped[(x: ConvertibleToInt = ..., /), int, (x: str), int]

The longer the function signature the uglier it gets. _Wrapped is an indicator that functools.update_wrapper was used which sets in this case foo.__wrapped__ = int as well as other attributes from int to foo. More precisely, but depending on the python version: '__module__', '__name__', '__qualname__', '__doc__', '__annotations__' and all entries from __dict__.

This is sometimes nice to know, but sometimes inconvenient and you would like to keep the original signature.

In practice you have two different cases 1) keep the signature identical 2) inject additional arguments.

For 1) You can use a bound Callable TypeVar or the new 3.12+ syntax:

Python <3.12

from typing import TypeVar, Callable
from functools import wraps

CT = TypeVar("CT", bound=Callable[..., Any])

def args_as_ints(f : CT) -> CT:
    """
    This tells the type-checker that the returned
    value is equivalent to the input value.
    """
    @wraps(f)
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g
...

Python 3.12+

from typing import Any, Callable
from functools import wraps

# Any is narrowed by the int from funny_function.
def args_as_ints[T: Callable[..., Any]](func: T) -> T:
    @wraps(func)
    def wrapper(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)

    return wrapper

...

reveal_type(funny_function)
# funny_function" is "(x: Any, y: Any, z: Any = 3) -> int"

2 ) Extend the signature

An alternative to keep the type-hint identical simple to use the decorator a bit differently and use a ParamSpec und functools.update_wrapper

from typing import TypeVar, Callable, ParamSpec, Concatenate
from functools import update_wrapper

T = TypeVar("T")
P = ParamSpec("P")


def args_as_ints(f : Callable[P, T]): # -> Callable[Concatenate[int, P], T]:
    # In this case you can also replace it with "int" to be explicit and not use
    # the signature of f, i.e. args: int
    def g(special_value=1, *args: P.args, **kwargs: P.kwargs) -> T:
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    update_wrapper(g, f)
    return g

ConvertibleToInt : TypeAlias = Any # you can do this properly

@args_as_ints
def funny_function(x : "ConvertibleToInt", 
                   y: "ConvertibleToInt", 
                   z: "ConvertibleToInt"=3) -> int:
    """Computes x*y + 2*z"""
    return x*y + 2*z

reveal_type(funny_function)
# "funny_function" is "(special_value: int = ... x: Any, y: Any, z: Any = ...) -> int"

At runtime this is equivalent to @wraps.

However there is one Pro and one con argument:
Pro: This is proper explicit-typing, it helps to understand the code and you can avoid errors.
Con: You likely loose your default arguments, as you see z has no default value anymore. An explicit annotation with -> Callable[Concatenate[int, P], T]: removes the information that the first parameter is named special_value. Implicit type/signature tracing can keep this information and also does not use _Wrapped. You loose the implicit tracing when you annotate f or the return type in the decorator.

Upvotes: 0

Dogeek

Reputation: 302

from inspect import signature


def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    sig = signature(f)
    g.__signature__ = sig
    g.__doc__ = f.__doc__
    g.__annotations__ = f.__annotations__
    g.__name__ = f.__name__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

I wanted to add that answer (since this shows up first in google). The inspect module is able to fetch the signature of a function, so that it can be preserved in decorators. But that's not all. If you want to modify the signature, you can do so like this :

from inspect import signature, Parameter, _ParameterKind


def foo(a: int, b: int) -> int:
    return a + b

sig = signature(foo)
sig._parameters = dict(sig.parameters)
sig.parameters['c'] = Parameter(
    'c', _ParameterKind.POSITIONAL_OR_KEYWORD, 
    annotation=int
)
foo.__signature__ = sig

>>> help(foo)
Help on function foo in module __main__:

foo(a: int, b: int, c: int) -> int

Why would you want to mutate a function's signature ?

It's mostly useful to have adequate documentation on your functions and methods. If you're using the *args, **kwargs syntax and then popping arguments from kwargs for other uses in your decorators, that keyword argument won't be properly documented, hence, modifying the signature of the function.

Upvotes: 3

Yilmaz

Reputation: 49281

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

this fixes name and documentation. to preserve the function signature, wrap is used exactly at same location as g.__name__ = f.__name__, g.__doc__ = f.__doc__.

the wraps itself a decorator. we pass the closure-the inner function to that decorator, and it is going to fix up the metadata. BUt if we only pass in the inner function to wraps, it is not gonna know where to copy the metadata from. It needs to know which function's metadata needs to be protected. It needs to know the original function.

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g=wraps(f)(g)
    return g

wraps(f) is going to return a function which will take g as its parameter. And that is going to return closure and will assigned to g and then we return it.

Upvotes: 1

smarie

Reputation: 5156

As commented above in jfs's answer ; if you're concerned with signature in terms of appearance (help, and inspect.signature), then using functools.wraps is perfectly fine.

If you're concerned with signature in terms of behavior (in particular TypeError in case of arguments mismatch), functools.wraps does not preserve it. You should rather use decorator for that, or my generalization of its core engine, named makefun.

from makefun import wraps

def args_as_ints(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("wrapper executes")
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# wrapper executes
# 22

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

funny_function(0)  
# observe: no "wrapper executes" is printed! (with functools it would)
# TypeError: funny_function() takes at least 2 arguments (1 given)

Upvotes: 4

Tim

Reputation: 13058

This is solved with Python's standard library functools and specifically functools.wraps function, which is designed to "update a wrapper function to look like the wrapped function". It's behaviour depends on Python version, however, as shown below. Applied to the example from the question, the code would look like:

from functools import wraps

def args_as_ints(f):
    @wraps(f) 
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

When executed in Python 3, this would produce the following:

>>> funny_function("3", 4.0, z="5")
22
>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Its only drawback is that in Python 2 however, it doesn't update function's argument list. When executed in Python 2, it will produce:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

Upvotes: 32

jfs

Reputation: 414199

Install decorator module:
```
$ pip install decorator
```

Adapt definition of args_as_ints():

import decorator

@decorator.decorator
def args_as_ints(f, *args, **kwargs):
    args = [int(x) for x in args]
    kwargs = dict((k, int(v)) for k, v in kwargs.items())
    return f(*args, **kwargs)

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

print funny_function("3", 4.0, z="5")
# 22
help(funny_function)
# Help on function funny_function in module __main__:
# 
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

Python 3.4+

functools.wraps() from stdlib preserves signatures since Python 3.4:

import functools


def args_as_ints(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# 22
help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

functools.wraps() is available at least since Python 2.5 but it does not preserve the signature there:

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(*args, **kwargs)
#    Computes x*y + 2*z

Notice: *args, **kwargs instead of x, y, z=3.

Upvotes: 108

macm

Reputation: 2019

Second option:

Install wrapt module:

$ easy_install wrapt

wrapt have a bonus, preserve class signature.


import wrapt
import inspect

@wrapt.decorator
def args_as_ints(wrapped, instance, args, kwargs):
    if instance is None:
        if inspect.isclass(wrapped):
            # Decorator was applied to a class.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to a function or staticmethod.
            return wrapped(*args, **kwargs)
    else:
        if inspect.isclass(instance):
            # Decorator was applied to a classmethod.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to an instancemethod.
            return wrapped(*args, **kwargs)


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x * y + 2 * z


>>> funny_function(3, 4, z=5))
# 22

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Upvotes: 6

Dzinx

Reputation: 57794

There is a decorator module with decorator decorator you can use:

@decorator
def args_as_ints(f, *args, **kwargs):
    args = [int(x) for x in args]
    kwargs = dict((k, int(v)) for k, v in kwargs.items())
    return f(*args, **kwargs)

Then the signature and help of the method is preserved:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

EDIT: J. F. Sebastian pointed out that I didn't modify args_as_ints function -- it is fixed now.

Upvotes: 9

Brian

Reputation: 119221

Take a look at the decorator module - specifically the decorator decorator, which solves this problem.

Upvotes: 8