What are Python metaclasses useful for?

Question

What can be done with metaclasses that can't be in any other way?

Alex Martelli told that there are tasks that can't be achieved without metaclasses here Python metaclasses vs class decorators I'd like to know which are?

Answer 1

Two things to keep in mind:

• You probably don't need them. They sound cool, but most people don't actually need them.

• Python supports dynamic classes via type(), multiple inheritance and method chaining via super().foo(). You can even add a function to a class and it will get bound to it at run time.

So, most of the time metaclasses are somewhat overkill.

However, one thing I found they can be very useful for is to assemble behavior from the attributes - not just the methods - of multiple classes.

For example, a validation framework that puts together multiple requirements:

• An http response should have

  • a status code - value to be specified in a subclass
  • content type - value to be specified in a subclass

• Furthermore, a html response checking class on which success is expected should:

  • expect 200 in that status code. html in the content type
  • expect a title - value to be specified... elsewhere

• The test instance would then only need specify the exact title.

A metaclass system can assemble all these requirements together so that your unittest only has to specify the title and the class "knows" it has to scrape the page for a <title>, check the content type and check the response code.

Here's a much more modest attempt at an example of using metaclasses for this. We are printing letters which have various header templates that get assembled together. The footer could use the same mechanism, but instead uses mixins a la Python’s super() considered super! . These 2 approaches - assembling class-level attributes and chaining methods other than __init__ via super work well together.

def get_annos(v):
    return getattr(v, "__annotations__", {})

class MailMeta(type):

    def get_annos(cls: type, ancestors_in: list[type]):
        res = {}
        for acls in ancestors_in:
            res.update(**get_annos(acls))
        return res

    def _calc_annotations(cls, li_bases):
        """ concatenate the annotations for the class and ancestors """
        annos = cls.get_annos(li_bases + [cls])
        cls.annos = annos

    def _concat_template(cls, varname, bases : list[type]):
        """this glues together some templates
        assembling complex attributes from multiple classes is 
        something metaclasses are uniquely good at.
        how to do this can be tricky though!  "Unordered" attributes
        like sets or dictionaries can be easier but use `copy` 
        liberally.      
        """

        li = []
        cls_var = f"cls_{varname}"
        for acls in bases:
            v = getattr(acls, cls_var, "")
            if not v:
                continue
            if v in li:
                continue

            li.append(v)

        all_ = "".join(li)
        setattr(cls, varname, all_)

    def __init__(cls, name, bases, attrs, **kwargs):
        """assemble together some of the attributes from the mro"""

        li_bases2current = [
            v for v in list(reversed(cls.mro())) if not v is object and not v is cls
        ]

        cls._calc_annotations(li_bases2current)
        cls._concat_template("header", [cls] + list(reversed(li_bases2current)) )

        super().__init__(name, bases, attrs)
        

class Mail(metaclass=MailMeta):
    """ specify some base behavior"""
    name : str  # mail always need to state who it is for
    salutation : str 

    cls_header : str = ""
    template : str
    footer : str = ""

    def _set_annotations(self, **kwargs):
        """"""
        undefined = object()

        for attrname in self.annos.keys():
            v = kwargs.get(attrname,undefined)
            if v is not undefined:
                setattr(self, attrname, v)
            else:
                #maybe its set on the classes
                v = getattr(self, attrname,undefined)
                if v is undefined:
                    raise ValueError(f"{attrname} is undefined")

    def __init__(self, **kwargs):
        """ track which variables we were expecting"""
        self._set_annotations(**kwargs)
        self.fulltemplate = "\n".join([self.header,self.template,self.get_footer()])

    def get_footer(self):
        return f"\n{self.footer}"

    def print(self):
        print(f"\n{'-' * 40}\n{self.__class__.__name__}:\n")

        di = dict(**vars(self.__class__))
        di.update(**vars(self))
        print(self.fulltemplate % (di))        


class Email:
    cls_header = "Dear %(salutation)s %(name)s,"
    cls_footer = "Sincerely,"

class Spam(Email, Mail):
    amount : int
    template = "You have won a prize of $%(amount)s"

class Signature:
    t_footer = "%(from_)s"
    from_ : str

    def get_footer(self):
        return super().get_footer() + self.t_footer % vars(self)

class TrustedSpam(Signature, Spam):
    "Trusted cuz signed"

    template = "You have won $%(amount)s.  This is a once in a lifetime opportunity."

    def get_footer(self):
        return super().get_footer() + "\nPlease trust me"

class SnailMailTrustedSpamForDoctors(TrustedSpam):
    "Doctors like to be called Dr."

    salutation = "Dr."
    cls_header = "101 Honest Fellows Street\n\n"


try:
    mail = Mail(saluation="Ms.", template="")
except (ValueError,) as exc: 
    print(f"ooops.  forgot `name`.              All good: {exc=} ✅")

try:
    mail = Spam(salutation="Ms.", name="Johnson")
except (Exception,) as exc: 
    print(f"ooops.  forgot `amount` on a Spam.  All good: {exc=} ✅")


mail = Spam(salutation="Ms.", amount=2000, name="Johnson")
mail.print()

mail = TrustedSpam(salutation="Ms.", amount=2000, name="Johnson", from_="Joe Trustworthy")
mail.print()

mail = SnailMailTrustedSpamForDoctors(amount=10000, name="Gullible", from_="Joe Trustworthy")
mail.print()

And the output:

ooops.  forgot `name`.              All good: exc=ValueError('name is undefined') ✅
ooops.  forgot `amount` on a Spam.  All good: exc=ValueError('amount is undefined') ✅

----------------------------------------
Spam:

Dear Ms. Johnson,
You have won a prize of $2000



----------------------------------------
TrustedSpam:

Dear Ms. Johnson,
You have won $2000.  This is a once in a lifetime opportunity.

Joe Trustworthy
Please trust me

----------------------------------------
SnailMailTrustedSpamForDoctors:

101 Honest Fellows Street

Dear Dr. Gullible,
You have won $10000.  This is a once in a lifetime opportunity.

Joe Trustworthy
Please trust me

Answer 2

Add extra flexibility to your programming:

But according to this Metaclass programming in Python you might not need them ( yet )

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).

-- Python Guru Tim Peters

Answer 3

The first commentator states that the use of metaclasses here was 'a gemstone' that helped him track down unexpected errors that had been occurring for (many) 'days'.

Answer 4

Metaclasses are indispensable if you want to have class objects (as opposed to instances of class objects) equipped with "special customized behavior", since an object's behavior depends on special methods on the type of the object, and a class object's type is, exactly a synonym for, the metaclass.

For example, if you want a class object X such that "print X" emits "Time is now 8:46am" (at 8:46 am, or, more generally, the current time) this must mean that type(x) (AKA X's metaclass) has a special custom __str__ method -- and similarly (with the various applicable special-methods) if you want to give meaning to expressions such as X + Y where X and Y are both class objects, or X[23] (where X, again, is a class object), and so forth.

Most other customization tasks are now (in Python 2.6 or better) easier to implement with a class decorator, which can alter a class object right after the end of the class statement. There are a few more cases where this is not feasible because the alterations must be made very early on if they are to have any effect (e.g., setting or altering __slots__).

In Python 3, metaclasses gain one extra little bit of usefulness: a metaclass can now optionally specify the mapping object to be populated during the execution of the class statement's body (by default, it's a normal dict). This allows the order of name bindings in the class body to be preserved and used (while the normal dict loses order), which is sometimes nice when the class must have "fields" in a certain specific order (e.g. to map 1:1 onto a C struct, a row in a CSV file or DB table, and the like) -- in Python 2.* this had to be redundantly specified (typically with an extra class attribute that's a sequence and thus does preserve order), and this feature of Python 3 metaclasses allows the redundancy to be removed.

Answer 5

I use metaclasses with some frequency, and they're an extremely powerful tool to have in the toolbox. Sometimes your solution to a problem can be more elegant, less code, with them than without.

The thing I find myself using metaclasses for most often, is post-processing the class attributes during class creation. For example, setting a name attribute on objects where appropriate (like how the Django ORM might work):

class AutonamingType(type):
    def __init__(cls, name, bases, attrs):
        for k,v in attrs.iteritems():
            if getattr(v, '__autoname__', False):
                v.name = k

class Autonamer(object):
    __metaclass__ = AutonamingType

If you have this as a tool, and you're using a class which must know its name before it can do_something():

class Foo(object):
    __autoname__ = True
    def __init__(self, name=None):
        self.name = name
    def do_something(self):
        if self.name is None:
            raise ValueError('name is None')
        # now, do something

It can make the difference in the rest of your code between this:

class Bar(object):
    myfoo1 = Foo('myfoo1')
    myfoo2 = Foo('myfoo2')
    myfoo3 = Foo('myfoo3')

and this:

class Baaz(Autonamer):
    myfoo1 = Foo()
    myfoo2 = Foo()
    myfoo3 = Foo()

Thus reducing duplication (and the chances that the variable name and the assigned name might get out-of-sync).

Answer 6

Take a look at Django sources - for example metaclasses are used there to generate models.

http://code.djangoproject.com/wiki/DynamicModels

Internally, Django uses metaclasses to create models based on a class you provide in your source code. Without getting into too many details, that means that rather than your classes being the actual models, Django receives a description of your class, which it uses to create a model in its place.

Answer 7

Probably there is nothing that can be done exclusively with metaclasses, but to some people (mine included) it's a interesting tool you can use. Just be careful not to abuse, as it can be tricky.

For example, I've used metaprogramming in a recent project. It was a OpenOffice calc sheet, which, using some pyUNO macros, generates some files with information. There was one sheet that presents to the user the information to fill, and the others can be used to describe the kind of elements and their properties. The user can then select the number of elements and the type of each, and generate the files. The macro will create a class via metaprogramming following the configuration on each sheet. Then, the user can instanciate each class and generate objects.

It could be done without metaprogramming, but to me seemed natural to use the metaprogramming capabilities to do it.

Answer 8

They are rarely needed, but come in useful in places where you want to add behavior to the basic behavior of an object -- compare Aspect Oriented Programming, or the instrumentation done in persistence frameworks like Hibernate.

For example, you may want a class which persists or logs each new object.