Python3.5: Class initialization using inheritance

Question

I recently stumbled across metaclasses in Python and decided to use them in order to simplify some features. (Using Python 3.5)

In a nutshell, I'm writing a module defining classes such as "components" that must be registered and initialized (I mean I need to initialize the actual class, not an instance).

I can register the class easily:

class MetaComponent(type):
    def __init__(cls, *args, **kargs):
        super().__init__(*args, **kargs)
        RegisterComponent(cls)

class BaseComponent(metaclass=MetaComponent):
    pass

class Component(BaseComponent):
    """This is the actual class to use when writing components"""

In this case, I'm registering the class of the component, as it allows me to refer to them later, without their actual reference.

But classes lack the ability to initialize themselves (at least in Python 3.5), and can cause some problems, like such:

class Manager(Component):
    SubManagers = []

    @classmethod
    def ListSubManagers(cls):
        for manager in cls.SubManagers:
            print(manager)

    @classmethod
    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

@Manager1.RegisterSubManager
class Manager2(Manager):
    pass

@Manager2.RegisterSubManager
class Manager3(Manager):
    pass

# Now the fun:
Manager1.ListSubManagers()

# Displays:
# > Manager2
# > Manager3

Now, this is a problem, because the idea was to have a unique list of sub-managers per manager. But the SubManager field is shared across each subclasses... So adding to one's list adds to every's. RIP.

So the next idea was to implement a kind of initializator:

class BaseManager(Component):
    @classmethod
    def classinit(cls):
        cls.SubManagers = []

But now, I need a way to call this method after the class creation. So lets do this with metaclasses again:

class MetaComponent(type):
    def __init__(cls, *args, **kargs):
        super().__init__(*args, **kargs):
        cls.classinit(**kargs)
        RegisterComponent(cls)

class BaseComponent(metaclass=MetaComponent):
    @classmethod
    def classinit(cls, **kargs):
        print('BASE', cls)

class Component(BaseComponent):
    @classmethod
    def classinit(cls, **kargs):
        super().classinit(**kargs) # Being able to use super() is the goal
        print('COMPONENT', cls)

I would consider myself done with this. Somewhat of an elegant way to do it IMO. The classinit() would be called from each class being created (unlike the 3.6 __init_subclass__ that is called on the parent). At least I liked it, until Python 3.5 cried in RuntimeError: super(): empty __class__ cell...

I read it was because I was calling a method from the metaclass's __init__ method and that although the class was created (hence my will to put code in __init__, to initialize something already created), it lacks this __class__ cell, at least at that moment...

I tried running the exact same code in Python 3.6, and it worked, so I guess something was wrong but got fixed...

My real questions are:

• Can we truly initialize classes with metaclasses in Python 3.5 ?
• Is there a way to avoid the use-restriction of super() in the initialization procedure?
• Why is it working in 3.6?
• If everything should fail, what would be the best course of action to still provide the class initialization, and allowing for super(...) calls? (Like do I need to refer to the super class explicitly ?)

Thanks for your help in advance.

EDIT:

The goal is to be able to derive components and be able to initialize each one's class relative to its parent, in an "easy" way:

class Manager(Component):
    def classinit(cls, **kargs):
        cls.SubManagers = []

    @classmethod
    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

@Manager.RegisterSubManager
class EventManager(Manager):
    def classinit(cls, **kargs):
        super().classinit(**kargs) # keep the old behaviour
        cls.Events = []

    # ...

@EventManager.RegisterSubManager
class InputManager(EventManager):
    def classinit(cls, **kargs):
        super().classinit(**kargs) # again, keep old behaviour
        cls.Inputs = []

    # use parts of EventManager, but define specialized methods
    # for input management

Managers are one concern, I have multiple concepts that depend on components and their ability to initialize their class.

Answer 1

Ok, so after some experiments, I managed to provide a "fix" in order to allow for class initialization, allowing the use of super().

First off, the module to "fix" the initialization method:

# ./PythonFix.py
import inspect
import types

def IsCellEmpty(cell):
    """Lets keep going, deeper !"""
    try:
        cell.cell_contents
        return False
    except ValueError:
        return True

def ClosureFix(cls, functionContainer):
    """This is where madness happens.
    I didn't want to come here. But hey, lets get mad.
    Had to do this to correct a closure problem occuring in
     Python < 3.6, joy.
    Huge thanks: https://stackoverflow.com/a/4885951/7983255
    """

    # Is the class decorated with @classmethod somehow
    isclassmethod = inspect.ismethod(functionContainer) and functionContainer.__self__ is cls
    if isclassmethod:
        function = functionContainer.__func__
    else:
        function = functionContainer

    # Get cells and prepare a cell holding ref to __class__
    ClosureCells = function.__closure__ or ()
    ClassCell_Fix = (lambda: cls).__closure__[0]

    # Shortcut
    c = function.__code__
    HasClassFreevar = '__class__' in c.co_freevars
    HasEmptyCells = any(IsCellEmpty(cell) for cell in ClosureCells)
    if HasClassFreevar and not HasEmptyCells: # No fix required.
        return classmethod(function)

    Freevars_Fixed = c.co_freevars
    Closure_Fixed = ClosureCells

    if not HasClassFreevar:
        Freevars_Fixed += ('__class__',)
        Closure_Fixed += (ClassCell_Fix,)

    elif HasEmptyCells: # This is silly, but for what I'm doing its ok.
        Closure_Fixed = tuple(ClassCell_Fix if IsCellEmpty(cell) else cell for cell in ClosureCells)

    # Now the real fun begins
    PyCode_fixedFreevars = types.CodeType(
        c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
        c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
        c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
        c.co_lnotab, Freevars_Fixed, c.co_cellvars
    )

    # Lets fetch the last closure to add our __class__ fix
    FixedFunction = types.FunctionType(
        PyCode_fixedFreevars, function.__globals__, function.__name__,
        function.__defaults__, Closure_Fixed
    )

    # Lets rewrap it so it is an actual classmethod (which it should be):
    return classmethod(FixedFunction)

And now, the component code:

class MetaComponent(type):
    def __init__(cls:type, *args, **kargs) -> None:
        super().__init__(*args, **kargs)
        if hasattr(cls, 'classinit'):
            cls.classinit = PythonFix.ClosureFix(cls, cls.classinit)
            cls.classinit(**kargs)
        RegisterComponent(cls)

    def classinit(cls:type, **kargs) -> None:
        """The default classinit method."""
        pass

class Component(metaclass=MetaComponent):
    """This class self registers, inherit from this"""

To be fair, I'm happy with it being done. Hope this helps someone wanting to initialize classes too (in a pre-Python3.6 env at least...).

Answer 2

If you want extra behavior for your Manager types, perhaps you want them to have their own, more refined metaclass, rather than just using the one they've inherited from Component. Try writing a MetaManager metaclass that inherits from MetaComponent. You can even move the class methods from Manager1 into the metaclass (where they become normal methods):

class MetaManager(MetaComponent):
    def __init__(cls, *args, **kwargs):
        super().__init__(*args, **kwargs)
        cls.SubManagers = [] # each class gets its own SubManagers list

    def ListSubManagers(cls):
        for manager in cls.SubManagers:
            print(manager)

    def RegisterSubManager(cls, manager):
        cls.SubManagers.append(manager)
        return manager

class Manager(Component, metaclass=MetaManager): # inherit from this to get the metaclass
    pass

class Manager1(Manager):
    pass

@Manager1.RegisterSubManager
class Manager2(Manager):
    pass

@Manager2.RegisterSubManager
class Manager3(Manager):
    pass

Answer 3

TL;DR - you will indeed have the RuntimeError: super(): empty __class__ cell... if you try to use an empty call to super from the metaclass __new__ or __init__ methods: at this stage the implicit "magic" variable __class__ that is internally used by super has not been created yet. (On verifying this, I just found out this has been fixed in Python 3.6 - that is: classmethods using parameterless super can be called from the metaclass's __init__ in Python 3.6, but yield this error in 3.5)

If that is the only thing in your way by now, just hardcode the call to the superclass method, like it was needed prior to the creation of super in Python. (Using the verbose form of super won't work as well).

--

Your second to last idea, of using classmethods as class decorators for registering, can be made to work by automatic creating a SubManagers attribute with the metaclass by using the a simple Python name-mangling to automatically create each manager class unique SubManagers attribute by inspecting one class' own namespace in its __dict__ (and it can be done without a metaclass as well)

Using metaclasses, just add these 2 lines at the end of your metaclass __init__:

if getattr(cls, "SubManagers") and not "SubManagers" in cls.__dict__:
    cls.SubManagers = []

If your class-decorator approach otherwise preclude metaclasses, you don't need to use a metaclass just for that - change your register method to perform the above "own" submanager list creation:

@classmethod
def RegisterSubManager(cls, manager):
   if not "SubManagers" in cls.__dict__:
       cls.SubManagers = []
   cls.SubManagers.append(manager)