Smart Pointer casting in Boost::Python

Question

I try to create Python bindings for an existing library using Boost::Python. The library uses custom smart pointers (called SmartPointer in the following examples). There are also two classes, Base and Derived (that inherits from Base).

Problems arise when I want to call a function expecting a SmartPointer<Derived> with a SmartPointer<Base> as an argument.

Is there a way to tell Boost::Python to try to "downcast" the SmartPointer<Base> to a SmartPointer<Derived> in such a case? I know that this "downcast" may fail, but it would add a lot of convenience.

Below is a minimal code example: (depending on your system you can compile it with g++ code.cpp -shared -o example.so -fPIC -I/usr/include/python3.2mu -lboost_python3 -lpython3.2mu)

#include <boost/python.hpp>
#include <iostream>

// ******** code to wrap ********
template <typename T>
class SmartPointer
{
public:
    explicit SmartPointer(T* p) : ptr(p) {}
    template <typename Y>
    explicit SmartPointer(Y* p) : ptr(static_cast<T*>(p)) {}
    template <typename Y>
    SmartPointer(SmartPointer<Y> const& src) : ptr(src.get()) {}
    T& operator*(void) const { return *ptr; }
    T* operator->(void) const { return ptr; }
    T* get(void) const { return ptr; }
protected:
    T* ptr;
};

class Base
{
public:
    virtual ~Base() {}
    virtual void say() const { std::cout << "Base" << std::endl; }
};

class Derived : public Base
{
public:
    virtual void say() const { std::cout << "Derived" << std::endl; }
    static SmartPointer<Base> create_base() { return SmartPointer<Base>(new Derived()); }
};

// ******** test functions ********
void test_basedirect(Base const& d) {
    d.say();
}

void test_basepointer(SmartPointer<Base> const& p) {
    p->say();
}

void test_deriveddirect(Derived const& d) {
    d.say();
}

void test_derivedpointer(SmartPointer<Derived> const& p) {
    p->say();
}

// ******** Boost::Python wrapping code ********
template <typename T>
T* get_pointer(SmartPointer<T> const& p) {
    return p.get();
}

namespace boost { namespace python {
    template <typename T>
    struct pointee<SmartPointer<T> > {
        typedef T type;
    };
}}

BOOST_PYTHON_MODULE(example) {
    using namespace boost::python;
    class_<Base, SmartPointer<Base>, boost::noncopyable>("Base", init<>())
        .def("say", &Base::say)
    ;
    class_<Derived, SmartPointer<Derived>, bases<Base>, boost::noncopyable>("Derived", init<>())
        .def("say", &Derived::say)
        .def("create_base", &Derived::create_base)
    ;
    def("test_basedirect", test_basedirect);
    def("test_basepointer", test_basepointer);
    def("test_deriveddirect", test_deriveddirect);
    def("test_derivedpointer", test_derivedpointer);
    implicitly_convertible<SmartPointer<Derived>, SmartPointer<Base> >();
}

and a Python session showing the failing call to the function expecting SmartPointer<Derived> as its parameter:

>>> from example import *
>>> d = Derived.create_base()
>>> test_basedirect(d)
Derived 
>>> test_basepointer(d)
Derived 
>>> test_deriveddirect(d)
Derived 
>>> test_derivedpointer(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    example.test_derivedpointer(Derived)
did not match C++ signature:
    test_derivedpointer(SmartPointer<Derived>)
>>> 

Answer 1

Assuming one cannot change Derived::create_base() to return SmartPointer<Derived>, which would allow for implicit conversions to be handled elsewhere, then conversions are still possible by explicitly registering a converter for SmartPointer<Derived>. When a Python object is being passed to C++ , Boost.Python will look within its registry to locate any converter that can construct the necessary C++ object. In this case, indirection is required, as the conversion occurs on the HeldTypes for the exposed C++ classes. The conversion steps for Derived's HeldType are:

1. If a Python object contains both Derived and SmartPointer<Base> C++ objects, then continue with the conversion.
2. Extract SmartPointer<Base> from the Python object.
3. Invoke custom function that constructs SmartPointer<Derived> from SmartPointer<Base>.

---

Here is a complete example, based on the original code:

#include <iostream>
#include <boost/python.hpp>
#include <boost/static_assert.hpp>

template <typename T>
class SmartPointer
{
public:
  explicit SmartPointer(T* p) : ptr(p) {}
  template <typename Y>
  explicit SmartPointer(Y* p) : ptr(static_cast<T*>(p)) {}
  template <typename Y>
  SmartPointer(SmartPointer<Y> const& src) : ptr(src.get()) {}
  T& operator*(void) const { return *ptr; }
  T* operator->(void) const { return ptr; }
  T* get(void) const { return ptr; }
protected:
  T* ptr;
};

class Base
{
public:
  virtual ~Base() {}
  virtual void say() const { std::cout << "Base" << std::endl; }
};

class Derived
  : public Base
{
public:
  virtual void say() const
  {
    std::cout << "Derived: " << this << std::endl;
  }

  static SmartPointer<Base> create_base()
  {
    return SmartPointer<Base>(new Derived());
  }
};

class OtherDerived
  : public Base
{
public:
  virtual void say() const
  {
    std::cout << "OtherDerived: " << this << std::endl;
  }

  static SmartPointer<Base> create_base()
  {
    return SmartPointer<Base>(new OtherDerived());
  }
};

void test_basedirect(Base const& d)                      { d.say();  }
void test_basepointer(SmartPointer<Base> const& p)       { p->say(); }
void test_deriveddirect(Derived const& d)                { d.say();  }
void test_derivedpointer(SmartPointer<Derived> const& p) { p->say(); }

// Boost.Python wrapping code.
template <typename T>
T* get_pointer(SmartPointer<T> const& p)
{
  return p.get();
}

namespace boost {
namespace python {

template <typename T>
struct pointee<SmartPointer<T> >
{
  typedef T type;
};

} // namespace python
} // namespace boost

namespace detail {
// @brief Construct Source from Target.
template <typename Source,
          typename Target>
Source construct_helper(Target& target)
{
  // Lookup the construct function via ADL.  The second argument is
  // used to:
  // - Encode the type to allow for template's to deduce the desired
  //   return type without explicitly requiring all construct functions
  //   to be a template.
  // - Disambiguate ADL when a matching convert function is declared
  //   in both Source and Target's enclosing namespace.  It should
  //   prefer Target's enclosing namespace.
  return construct(target, static_cast<boost::type<Source>*>(NULL));
}
} // namespace detail

/// @brief Enable implicit conversions between Source and Target types
///        within Boost.Python.
///
///        The conversion of Source to Target should be valid with
///        `Target t(s);` where `s` is of type `Source`.
///
///        The conversion of Target to Source will use a helper `construct`
///        function that is expected to be looked up via ADL.
///
///        `Source construct(Target&, boost::type<Source>*);`
template <typename Source,
          typename Target>
struct two_way_converter
{
  two_way_converter()
  {
    // Enable implicit source to target conversion.
    boost::python::implicitly_convertible<Source, Target>();

    // Enable target to source conversion, that will use the convert
    // helper.
    boost::python::converter::registry::push_back(
      &two_way_converter::convertible,
      &two_way_converter::construct,
      boost::python::type_id<Source>()
    );
  }

  /// @brief Check if PyObject contains the Source pointee type.
  static void* convertible(PyObject* object)
  {
    // The object is convertible from Target to Source, if:
    // - object contains Target.
    // - object contains Source's pointee.  The pointee type must be
    //   used, as this is the converter for Source.  Extracting Source
    //   would cause Boost.Python to invoke this function, resulting
    //   infinite recursion.
    typedef typename boost::python::pointee<Source>::type pointee;
    return boost::python::extract<Target>(object).check() &&
           boost::python::extract<pointee>(object).check()
        ? object
        : NULL;
  }

  /// @brief Convert PyObject to Source type.
  static void construct(
    PyObject* object,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    namespace python = boost::python;

    // Obtain a handle to the memory block that the converter has allocated
    // for the C++ type.
    typedef python::converter::rvalue_from_python_storage<Source>
                                                            storage_type;
    void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

    // Extract the target.
    Target target = boost::python::extract<Target>(object);

    // Allocate the C++ type into the converter's memory block, and assign
    // its handle to the converter's convertible variable.  The C++ type
    // will be copy constructed from the return of construct function.
    data->convertible = new (storage) Source(
      detail::construct_helper<Source>(target));
  }
};

/// @brief Construct SmartPointer<Derived> from a SmartPointer<Base>.
template <typename Derived>
Derived construct(const SmartPointer<Base>& base, boost::type<Derived>*)
{
  // Assumable, this would need to do more for a true smart pointer.
  // Otherwise, two unrelated sets of smart pointers are managing the
  // same instance.
  return Derived(base.get());
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose Base.
  python::class_<Base, SmartPointer<Base>, boost::noncopyable>(
      "Base", python::init<>())
    .def("say", &Base::say)
    ;

  // Expose Derived.
  python::class_<Derived, SmartPointer<Derived>, 
       python::bases<Base>, boost::noncopyable>(
         "Derived", python::init<>())
    .def("say", &Derived::say)
    .def("create_base", &Derived::create_base)
    .staticmethod("create_base");
    ;

  // Expose OtherDerived.
  python::class_<OtherDerived, SmartPointer<OtherDerived>, 
      python::bases<Base>, boost::noncopyable>(
        "OtherDerived", python::init<>())
    .def("say", &OtherDerived::say)
    .def("create_base", &OtherDerived::create_base)
    .staticmethod("create_base");
    ;

  // Expose Test functions.
  python::def("test_basedirect",          &test_basedirect);
  python::def("test_basepointer",         &test_basepointer);
  python::def("test_deriveddirect",       &test_deriveddirect);
  python::def("test_derivedpointer",      &test_derivedpointer);

  // Enable conversions between the types.
  two_way_converter<SmartPointer<Derived>,      SmartPointer<Base> >();
  two_way_converter<SmartPointer<OtherDerived>, SmartPointer<Base> >();
}

And its usage:

>>> from example import *
>>> d = Derived.create_base()
>>> print d
<example.Derived object at 0xb7f34b1c>
>>> test_basedirect(d)
Derived: 0x8f4de18
>>> test_basepointer(d)
Derived: 0x8f4de18
>>> test_deriveddirect(d)
Derived: 0x8f4de18
>>> test_derivedpointer(d)
Derived: 0x8f4de18
>>> 
>>> o = OtherDerived.create_base()
>>> print o
<example.OtherDerived object at 0xb7f34b54>
>>> test_basedirect(o)
OtherDerived: 0x8ef6dd0
>>> test_basepointer(o)
OtherDerived: 0x8ef6dd0
>>> test_derivedpointer(o)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    example.test_derivedpointer(OtherDerived)
did not match C++ signature:
    test_derivedpointer(SmartPointer<Derived>)

A few notes/comments about the implementation:

• two_way_converter assumes the type arguments are HeldTypes. Thus, boost::python::pointee<T>::type must be valid for the HeldTypes.
• two_way_converter allows users to enable their own custom strategy to construct a Source from Target by declaring a Source construct(Target, boost::type<Source>*) function in Target's enclosing namespace. The second argument's value is not meaningful, as it will always be NULL.
• two_way_converter::convertible()'s criteria needs to be thorough enough that Source construct(Target) shall not fail.

---

Also, test_deriveddirect() works because Boost.Python performs introspection when creating a Python object from a C++ object. When classes are exposed, Boost.Python constructs a graph with type information. When a C++ object passes to Python, Boost.Python will transverse the graph until the appropriate Python type is found based on the C++ object's dynamic type. Once found, the Python object is allocated and holds the C++ object or its HeldType.

In the example code, Boost.Python know that Base is held by SmartPointer<Base> and Derived is derived from Base. Hence, Derived may be held by SmartPointer<Base>. When SmartPointer<Base> is passed to Python, Boost.Python gets a pointer to the C++ object via the get_pointer() function. The static type Base is used to find a node in the graph, then traversal occurs attempting to identify the dynamic type of the C++ object. This results in Boost.Python creating an example.Base object when SmartPointer<Base> points to an object with a dynamic type of Base, and creating an example.Derived object when SmartPointer<Base> points to an object with a dynamic type of Derived.

>>> d = Derived.create_base()
>>> print d
<example.Derived object at 0xb7f34b1c>

As Boost.Python knows d contains a C++ Derived object, the call to test_deriveddirect(d) works.