Retrieving a Python type back from c++

Question

This question is really an extension of the following two questions:

1. How can I implement a C++ class in Python, to be called by C++?
2. Swig downcasting from Base* to Derived*

Suppose that I have the following c++ classes (simplified), which I am exposing to Python using SWIG:

struct Component
{
    virtual void update(double dt);
}

struct DerivedComponent : public Component
{
    void update(double dt) { std::cout << "DerivedComponent::update()" << std::endl; }
    void f() { std::cout << "DerivedComponent::f()" << std::endl; }
}

class Entity
{
public:
    Component* component(const std::string& class_name)
    {
        return m_components[class_name];
    }

    Component* create_component(const std::string& class_name)
    {
        // Creates a new Component, possibly using the method described in (1),
        // adds it to m_components and returns the component.
        // This method can also instantiate subclasses of Component that were
        // defined in c++.
    }

private:
    std::unordered_map<std::string, Component*> m_components;
}

Now, in Python, I define a class that inherits from Component:

class PythonDerivedComponent(Component):
    def __init__(self):
        Component.__init__(self)
    def update(self, dt):
        print("DerivedComponent::update(" + str(dt) + ")")
    def g()
        print("DerivedComponent::g()")

I have got to the stage where I can add components to entities. Also, using the method described by Flexo in (2), I can retrieve the derived component type from create_component() for when the component is defined in c++:

e = Entity()
c = e.create_component("DerivedComponent") # Has type <class 'module.DerivedComponent'>
c.f() # Prints "DerivedComponent::f()" as expected.

Now, my question is whether it is possible to get the derived component type from create_component() for when the component is defined in Python? Of course, I only need to be able to do this from within Python.

e = Entity("Player")
e.create_component("PythonDerivedComponent")

# Somewhere else...
e = get_entity("Player")
c = e.component("PythonDerivedComponent") # Has type <class 'module.Component'>
c.g() # Can't call this function.

Answer 1

In order to make a complete working demo I had to expand on your header file slightly, mine ended up looking like:

#ifndef TEST_HH
#define TEST_HH

#include <map>
#include <functional>
#include <iostream>
#include <string>

struct Component
{
    virtual void update(double dt) = 0;
    virtual ~Component() {}
};

struct DerivedComponent : public Component
{
    void update(double) { std::cout << "DerivedComponent::update()" << std::endl; }
    void f() { std::cout << "DerivedComponent::f()" << std::endl; }

    static DerivedComponent *create() { 
        return new DerivedComponent;
    }
};

class Entity
{
public:
    Component* component(const std::string& class_name)
    {
        return m_components[class_name];
    }

    Component* create_component(const std::string& class_name)
    {
        // Creates a new Component, possibly using the method described in (1),
        // adds it to m_components and returns the component.
        // This method can also instantiate subclasses of Component that were
        // defined in c++.
        Component *result = nullptr;
        if (m_components[class_name]) {
            result = m_components[class_name];
        }
        else if (m_registry[class_name]) {
            result = m_registry[class_name]();
            m_components[class_name] = result;
        }
        return result; // Or raise an exception if null?
    }

    void register_component(const std::string& class_name, std::function<Component*()> creator) {
        m_registry[class_name] = creator;
    }
private:
    std::map<std::string, Component*> m_components;
    std::map<std::string, std::function<Component*()> > m_registry;
};

inline void register_builtins(Entity& e) {
    e.register_component("DerivedComponent", DerivedComponent::create);    
}

#endif

Mainly that fixed a few syntax errors and added a registry of types, with std::function objects that know how to create instances.

We're building on the typemaps from the two questions you referenced, so I won't talk about that part much in this answer, except to say that in order to make your example Python class work we had to use directors (otherwise it's permanently abstract) and the 'out' typemap from the previous question has been modified to be applied in more places as well as adding the extra functionality. (Note that this assumes the string with class_name will be arg2 always. You could add a function to the base class that returns this name which would remove the need for this assumption)

There are two main tricks we need to make this work:

1. We need to implement a 'Python aware' version of Component::register_component. In this instance I've implemented it with a C++11 lambda function, which retains a reference to the type of the product it is going to produce. (Note that my example would leak these types, because it never decrements the reference counter. You should use a smart pointer if that's a problem for your usage).
2. We need to keep a hold of the real PyObject that was created when constructing our Python derived types. There are several ways you could do that, for example with a global map of std::map<Component*,PyObject*>, or by actually adding it within your Entity class in test.hh. I dislike globals and worked on the assumption that you don't want to end up mixing concerns of the Python interface with the C+++ implementation. As a result I opted to add another, intermediate abstract class that inherits from Component and serves only to remember what the PyObject that implements it really was.

There's a possible fringe benefit to adding the intermediate PythonComponent class, which is that you won't end up paying the cost of the SWIG directors for pure C++ derived types. (If you wanted to you could play games using %pythoncode and %rename to pretend to Python developers that they really are just using Component and not PythonComponent, but I've not done that here)

My SWIG interface file thus ended up looking like:

%module(directors=1) test

%{
#include "test.hh"
#include <cassert>
struct PythonComponent : Component {
    PyObject *derrived;
};
%}

%feature("director") PythonComponent;

%include <std_string.i>

// Note: this now gets applied to anything returning Component *
%typemap(out) Component * {
    const PythonComponent * const pycomp = dynamic_cast<PythonComponent*>($1);
    if (pycomp) {
        $result = pycomp->derrived;
        Py_INCREF($result);
    }
    else {
        const std::string lookup_typename = *arg2 + " *";
        swig_type_info * const outtype = SWIG_TypeQuery(lookup_typename.c_str());
        $result = SWIG_NewPointerObj(SWIG_as_voidptr($1), outtype, $owner);
    }
}

%include "test.hh"


struct PythonComponent : Component {
};

%extend Entity {
    void register_component(const std::string& class_name, PyObject *python_type) {
        assert(PyCallable_Check(python_type));
        Py_INCREF(python_type);
        $self->register_component(class_name, [python_type](){
            PyObject *pyinstance = PyObject_CallObject(python_type, NULL);
            void *result;
            const auto res = SWIG_ConvertPtr(pyinstance, &result,SWIGTYPE_p_PythonComponent, 0);
            if (!SWIG_IsOK(res)) {
                assert(false); // TODO: raise exception
            }
            const auto out = reinterpret_cast<PythonComponent *>(result);
            out->derrived = pyinstance;
            return out;
        });
    }
}

We used %extend to implement the overload of register_component. That overload would also allow us to control the registration of native C++ types from within Python with only minimal extra effort, I wrote an answer about that previously.

As far as the Python wrapper is concerned the PythonComponent type doesn't actually change Component at all. The detail of the retained reference is kept as an implementation detail.

With these mechanics in place all we need to do in order to make this work is to implement the new 'out' typemap that adds a dynamic_cast to figure out if the C++ type in the table is really a PythonComponent and if it is use the retained PyObject instead of doing a lookup for the SWIG types.

We compile with:

swig -py3 -c++ -python -Wall test.i
g++ -std=c++11 -Wall -Wextra test_wrap.cxx -I/usr/include/python3.4/ -lpython3.4m -shared -o _test.so

And I adapted your testcase to correct a few issues and call register_component:

from test import *

e = Entity()
register_builtins(e)

c = e.create_component("DerivedComponent") # Has type <class 'module.DerivedComponent'>
c.f() # Prints "DerivedComponent::f()" as expected.

class PythonDerivedComponent(PythonComponent):
    def update(self, dt):
        print("PythonDerivedComponent::update(" + str(dt) + ")")
    def g(self):
        print("PythonDerivedComponent::g()")

e.register_component("PythonDerivedComponent", PythonDerivedComponent)
e.create_component("PythonDerivedComponent")

c = e.component("PythonDerivedComponent")
print(type(c))
c.g() # Now works.

When we run it we see:

DerivedComponent::f()
<class '__main__.PythonDerivedComponent'>
PythonDerivedComponent::g(