python late binding - dynamically put locals in scope

Question

i have a function, m_chain, which refers to two functions bind and unit which are not defined. i want to wrap this function in some context which provides definitions for these functions - you can think of them as interfaces for which i want to dynamically provide an implementation.

def m_chain(*fns):
    """what this function does is not relevant to the question"""
    def m_chain_link(chain_expr, step):
        return lambda v: bind(chain_expr(v), step)
    return reduce(m_chain_link, fns, unit)

In Clojure, this is done with macros. what are some elegant ways of doing this in python? i have considered:

• polymorphism: turn m_chain into a method referring to self.bind and self.unit, whose implementations are provided by a subclass
• implementing the with interface so i can modify the environment map and then clean up when i'm done
• changing the signature of m_chain to accept unit and bind as arguments
• requiring usage of m_chain be wrapped by a decorator which will do something or other - not sure if this even makes sense

ideally, i do not want to modify m_chain at all, i want to use the definition as is, and all of the above options require changing the definition. This is sort of important because there are other m_* functions which refer to additional functions to be provided at runtime.

How do i best structure this so i can nicely pass in implementations of bind and unit? its important that the final usage of m_chain be really easy to use, despite the complex implementation.

edit: here's another approach which works, which is ugly as all hell because it requires m_chain be curried to a function of no args. but this is a minimum working example.

def domonad(monad, cmf):
    bind = monad['bind']; unit = monad['unit']
    return cmf()

identity_m = {
    'bind':lambda v,f:f(v),
    'unit':lambda v:v
}

maybe_m = {
    'bind':lambda v,f:f(v) if v else None,
    'unit':lambda v:v
}

>>> domonad(identity_m, lambda: m_chain(lambda x: 2*x, lambda x:2*x)(2))
8
>>> domonad(maybe_m, lambda: m_chain(lambda x: None, lambda x:2*x)(2))
None

Answer 1

Okay, here is my final answer to this question.

You need to be able to rebind some functions at least some of the time. Your hack, backing up the .__globals__ value and pasting in new values, is ugly: slow, non-thread-safe, and specific to CPython. I have thought about this, and there is no Pythonic solution that works this way.

In Python, you can rebind any function, but you have to do it explicitly, and some functions are not a good idea to rebind. For example, I love the builtins all() and any(), and I think it would be scary if you could stealthily rebind them and it would not be obvious.

You want some functions to be rebindable, and I don't think you need them all to be rebindable. So it would make perfect sense to mark the rebindable functions in some way. The obvious and Pythonic way to do this is to make them method functions of a class we can call Monad. You can use the standard variable name m for instances of Monad, and then when someone tries to read and understand their code, they will know that a function with a name like m.unit() is potentially rebindable via some other Monad instance being passed in.

It will be pure Python, and completely portable, if you obey these rules:

1. All functions must be bound in the monad. If you refer to m.bind() then "bind" must appear in the .__dict__ of the instance of Monad.
2. Functions using Monad must take a named argument m=, or for functions that will use the *args feature, must take a **kwargs argument and check it for a key named "m".

Here is an example of what I have in mind.

class Monad(object):
    def __init__(self, *args, **kwargs):
        # init from each arg.  Try three things:
        # 0) if it has a ".__dict__" attribute, update from that.
        # 1) if it looks like a key/value tuple, insert value for key.
        # 2) else, just see if the whole thing is a dict or similar.
        # Other instances of class Monad() will be handled by (0)
        for x in args:
            if hasattr("__dict__", x):
                self.__dict__.update(x.__dict__)
            else:
                try:
                    key, value = x
                    self.__dict__[key] = value
                except TypeError:
                    self.__dict__.update(x)
        self.__dict__.update(kwargs)


def __identity(x):
    return x

def __callt(v, f):
    return f(v)

def __callt_maybe(v, f):
    if v:
        return f(v)
    else:
        return None

m_identity = Monad(bind=__callt, unit=__identity)
m_maybe = Monad(bind=__callt_maybe, unit=__identity)

def m_chain(*fns, **kwargs):
    """what this function does is not relevant to the question"""
    m = kwargs.get("m", m_identity)
    def m_chain_link(chain_expr, step):
        return lambda v: m.bind(chain_expr(v), step)
    return reduce(m_chain_link, fns, m.unit)

print(m_chain(lambda x: 2*x, lambda x:2*x, m=m_identity)(2)) # prints 8
print(m_chain(lambda x: None, lambda x:2*x, m=m_maybe)(2)) # prints None

The above is clean, Pythonic, and should run just as well under IronPython, Jython, or PyPy as it does under CPython. Inside m_chain(), the expression m = kwargs.get("m", m_identity) tries to read out a specified monad argument; if one is not found, the monad is set to m_identity.

But, you might want more. You might want the Monad class to support only optionally overriding a function name; and you might be willing to stick with just CPython. Here is a trickier version of the above. In this version, when the expression m.some_name() is evaluated, if the Monad instance m does not have the name some_name bound in its .__dict__, it will look up some_name in the locals of the caller, and in the globals().

In this case, the expression m.some_name() means "m can override some_name but doesn't have to; some_name might not be in m, in which case some_name will be looked up as if it were not prefixed by m.". The magic is in the function .__getattr__(), which uses sys._getframe() to peek at the locals of the caller. .__getattr__() is only called when the local lookup fails, so we know that the Monad instance doesn't have name bound in .__dict__; so look at the locals belonging to the caller, using sys._getframe(1).f_locals; failing that, look in globals(). Just insert this into the class definition of Monad in the source code above.

def __getattr__(self, name):
    # if __getattr__() is being called, locals() were already checked
    d = sys._getframe(1).f_locals
    if name in d:
        return d[name]

    d = globals()
    if name in d:
        return d[name]

    mesg = "name '%s' not found in monad, locals, or globals" % name
    raise NameError, mesg

Answer 2

Python is already late bound. There's no need to do any work here:

def m_chain(*args):
    return bind(args[0])

sourcemodulename = 'foo'
sourcemodule = __import__(sourcemodulename)
bind = sourcemodule.bind

print m_chain(3)

Answer 3

here is how i ended up doing it. no idea if this is a good idea. but it lets me write my m_* functions totally independent of the implementation of unit/bind, and also totally independent of any implementation details of the way monads are done in python. the right things are just there in lexical scope.

class monad:
    """Effectively, put the monad definition in lexical scope.
    Can't modify the execution environment `globals()` directly, because
    after globals().clear() you can't do anything.
    """
    def __init__(self, monad):
        self.monad = monad
        self.oldglobals = {}

    def __enter__(self):
        for k in self.monad:
            if k in globals(): self.oldglobals[k]=globals()[k]
            globals()[k]=self.monad[k]

    def __exit__(self, type, value, traceback):
        """careful to distinguish between None and undefined.
        remove the values we added, then restore the old value only
        if it ever existed"""
        for k in self.monad: del globals()[k]
        for k in self.oldglobals: globals()[k]=self.oldglobals[k]


def m_chain(*fns):
    """returns a function of one argument which performs the monadic
    composition of fns."""
    def m_chain_link(chain_expr, step):
        return lambda v: bind(chain_expr(v), step)
    return reduce(m_chain_link, fns, unit)


identity_m = {
    'bind':lambda v,f:f(v),
    'unit':lambda v:v
}

with monad(identity_m):
    assert m_chain(lambda x:2*x, lambda x:2*x)(2) == 8


maybe_m = {
    'bind':lambda v,f:f(v) if v else None,
    'unit':lambda v:v
}

with monad(maybe_m):
    assert m_chain(lambda x:2*x, lambda x:2*x)(2) == 8
    assert m_chain(lambda x:None, lambda x:2*x)(2) == None


error_m = {
    'bind':lambda mv, mf: mf(mv[0]) if mv[0] else mv,
    'unit':lambda v: (v, None)
}

with monad(error_m):
    success = lambda val: unit(val)
    failure = lambda err: (None, err)

    assert m_chain(lambda x:success(2*x), lambda x:success(2*x))(2) == (8, None)
    assert m_chain(lambda x:failure("error"), lambda x:success(2*x))(2) == (None, "error")
    assert m_chain(lambda x:success(2*x), lambda x:failure("error"))(2) == (None, "error")


from itertools import chain
def flatten(listOfLists):
    "Flatten one level of nesting"
    return list(chain.from_iterable(listOfLists))

list_m = {
    'unit': lambda v: [v],
    'bind': lambda mv, mf: flatten(map(mf, mv))
}


def chessboard():
    ranks = list("abcdefgh")
    files = list("12345678")

    with monad(list_m):
        return bind(ranks, lambda rank:
               bind(files, lambda file:
                       unit((rank, file))))

assert len(chessboard()) == 64
assert chessboard()[:3] == [('a', '1'), ('a', '2'), ('a', '3')]

Answer 4

In Python, you can write all the code you want that refers to stuff that doesn't exist; to be specific, you can write code that refers to names that do not have values bound to them. And you can compile that code. The only problem will happen at run time, if the names still don't have values bound to them.

Here is a code example you can run, tested under Python 2 and Python 3.

def my_func(a, b):
    return foo(a) + bar(b)

try:
    my_func(1, 2)
except NameError:
    print("didn't work") # name "foo" not bound

# bind name "foo" as a function
def foo(a):
    return a**2

# bind name "bar" as a function
def bar(b):
    return b * 3

print(my_func(1, 2))  # prints 7

If you don't want the names to be just bound in the local name space, but you want to be able to fine-tune them per function, I think the best practice in Python would be to use named arguments. You could always close over the function arguments and return a new function object like so:

def my_func_factory(foo, bar):
    def my_func(a, b):
        return foo(a) + bar(b)
    return my_func

my_func0 = my_func_factory(lambda x: 2*x, lambda x:2*x)
print(my_func0(1, 2))  # prints 6

EDIT: Here is your example, modified using the above idea.

def domonad(monad, *cmf):
    def m_chain(fns, bind=monad['bind'], unit=monad['unit']):
        """what this function does is not relevant to the question"""
        def m_chain_link(chain_expr, step):
            return lambda v: bind(chain_expr(v), step)
        return reduce(m_chain_link, fns, unit)

    return m_chain(cmf)

identity_m = {
    'bind':lambda v,f:f(v),
    'unit':lambda v:v
}

maybe_m = {
    'bind':lambda v,f:f(v) if v else None,
    'unit':lambda v:v
}

print(domonad(identity_m, lambda x: 2*x, lambda x:2*x)(2)) # prints 8
print(domonad(maybe_m, lambda x: None, lambda x:2*x)(2)) # prints None

Please let me know how this would work for you.

EDIT: Okay, one more version after your comment. You could write arbitrary m_ functions following this pattern: they check kwargs for a key "monad". This must be set as a named argument; there is no way to pass it as a positional argument, because of the *fns argument which collects all arguments into a list. I provided default values for bind() and unit() in case they are not defined in the monad, or the monad is not provided; those probably don't do what you want, so replace them with something better.

def m_chain(*fns, **kwargs):
    """what this function does is not relevant to the question"""
    def bind(v, f):  # default bind if not in monad
        return f(v),
    def unit(v):  # default unit if not in monad
        return v
    if "monad" in kwargs:
        monad = kwargs["monad"]
        bind = monad.get("bind", bind)
        unit = monad.get("unit", unit)

    def m_chain_link(chain_expr, step):
        return lambda v: bind(chain_expr(v), step)
    return reduce(m_chain_link, fns, unit)

def domonad(fn, *fns, **kwargs):
    return fn(*fns, **kwargs)

identity_m = {
    'bind':lambda v,f:f(v),
    'unit':lambda v:v
}

maybe_m = {
    'bind':lambda v,f:f(v) if v else None,
    'unit':lambda v:v
}

print(domonad(m_chain, lambda x: 2*x, lambda x:2*x, monad=identity_m)(2))
print(domonad(m_chain, lambda x: None, lambda x:2*x, monad=maybe_m)(2))