Boost.Python: inheret from C++ object in python: Cannot add to C++ base class list

Question

I am trying to extend existing C++ objects in Python via inheritance. I can do this successfully and run virtual methods overridden in Python. When I however, try to add the python object to a list of pointers of the C++ Base object type(the Base object the python class has overridden), I get a type error: 'Attempting to append an invalid type'

I am sure this error is due to there begin no 'implicitly_convertible' functionality from derived* to base*. In C++, this would be defined as so: implicitly_convertible<[Derived_from_base],Base>();. Is it possible to define this in python?

How can I achieve this?

Here is sample code reproducing this behaviour.

C++

struct Base {
    virtual ~Base() {}
    virtual int f() = 0;
};
struct A {
    std::vector<Base*>& GetBaseList() { return m_base_List; }
    std::vector<Base*> m_base_List;
};
struct BaseWrap : Base, wrapper<Base> {
    int f() { return this->get_override("f")(); }
};

BOOST_PYTHON_MODULE(sandbox)
{
    class_<BaseWrap, Base*, boost::noncopyable>("Base", no_init)
        .def("f", pure_virtual(&Base::f));

    class_<A, A*>("A", init<>())
        .add_property("baseList", make_function(&A::GetBaseList, return_internal_reference<>()));

    //implicitly_convertible<[Derived_from_base]*,Base*>();
    class_<std::vector<Base*>>("BaseList").def(vector_indexing_suite<std::vector<Base*>>());
}

Python from sandbox import *

class derived(Base):
    def __init__(self):
        self.name = "test"
    def f(self):
        print("Hello Derived!")

d = derived()
d.f()          # Output: Hello Derived!

a = A()
a.baseList.append(d) # TypeError: Attempting to append an invalid type

Any help or ideas will be greatly appreciated.

Answer 1

The BaseList.append() function receives an argument with the right type; however, the argument has an inappropriate value. In Python, the derived initializer is not initializing the sandbox.Base part of its hierarchy. This results in the Boost.Python object not containing a C++ BaseWrap object. Hence, when BaseList.append() attempts to extract the C++ BaseWrap object, it fails and throws an error.

class derived(Base):
    def __init__(self):
        self.name = "test"
        # Base is not initialized.
    def f(self):
        print("Hello Derived!")

d = derived()
d.f() # `derived.f()` is resolved through Python's method-resolution-order.  
      # It is not invoking `BaseWrap::f()`.

a = A()
a.baseList.append(d) # d does not contain a BaseWrap object, so this throws.

To resolve the issue, explicitly invoke Base.__init__() within derived.__init__():

class derived(Base):
    def __init__(self):
        self.name = "test"
        Base.__init__(self)

However, attempting to do this will surface other problems with how BaseWrap is exposed:

• The sandbox.Base class must be constructible from Python, so the bindings cannot provide boost::python::no_init as its initializer specification. Generally, one would only want to use boost::python::no_init when the C++ objects are being explicitly instantiated from C++ and passed to Python, such as via factory functions.
• When T is BaseWrap, a HeldType of Base* fails to meet the requirements of HeldType. In particular, the HeldType either needs to be: BaseWrap, a class derived from BaseWrap, or a dereferenceable type for which boost::python::pointee<Base*>::type is BaseWrap or a class derived from BaseWrap. See the class_ specification for requirement details.

These can be resolved by exposing the class as follows:

namespace python = boost::python;
python::class_<BaseWrap, boost::noncopyable>("Base", python::init<>())
  .def("f", python::pure_virtual(&Base::f))
  ;

---

Here is a complete example demonstrating passing an object that derives from a C++ exposed class to a C++ vector exposed via the vector_indexing_suite:

#include <vector>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>

struct base
{
  virtual ~base() {}
  virtual int perform() = 0;
};

struct base_wrap: base, boost::python::wrapper<base>
{
  int perform() { return int(this->get_override("perform")()) - 10; }
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  python::class_<base_wrap, boost::noncopyable>("Base", python::init<>())
    .def("perform", python::pure_virtual(&base::perform))
    ;

  python::class_<std::vector<base*>>("BaseList")
    .def(python::vector_indexing_suite<std::vector<base*>>())
    ;

  python::def("do_perform", +[](base* object) {
    return object->perform();
  });
}

Interactive usage:

>>> import example
>>> class derived(example.Base):
...     def __init__(self):
...         self.name = "test"
...         example.Base.__init__(self)
...     def perform(self):
...         return 42
...       
>>> d = derived()
>>> base_list = example.BaseList()
>>> base_list.append(d)
>>> assert(len(base_list) == 1)
>>> assert(base_list[0].perform() == 42)
>>> assert(example.do_perform(base_list[0]) == 32)

With collections and pointers, there are often some caveats. In this case:

• The BaseList object does not have shared ownership of objects to which its elements refer. Be careful to guarantee that objects referenced by the container have a lifetime at least as long as the container itself. In the above example, if object d is deleted, then invoking base_list[0].perform() can result in undefined behavior.
• One cannot iterate over the base_list, as the iterator's value will attempt to perform a base*-to-Python conversion, which does not exists.

---

The above example also demonstrates the difference in function dispatching. If Python can directly invoke a method, it will do so using its own method-resolution mechanics. Note how base_list[0].perform() and example.do_perform(base_list[0]) return different values, as one gets dispatched through base_wrap::perform() which manipulates the result, and the other does not.

In the original code:

class derived(sandbox.Base):
    ...
    def f(self):
        print("Hello Derived!")

d = derived()
d.f()

As Python is aware of derived.f(), invoking d.f() will not get dispatched through BaseWrap::f(). If BaseWrap::f() had been invoked, it would have thrown because derived.f() returned None, which will fail to convert to an int:

struct BaseWrap : Base, wrapper<Base> {
    int f() { return this->get_override("f")(); }
                        // ^~~ returns a boost::python::object, faling to 
                        //     extract `int` will throw. 
};