Boost Python Exposing C++ class with constructor taking a std::list

Question

I have a class that looks like the following,

class MyClass
{
    MyClass( std::list<std::string> );

};

I tried exporting it to python using,

class_<MyClass, boost::noncopyable >("MyClass", init<std::list<std::string>>());

But i got a signature mismatch error,

did not match C++ signature:
__init__(_object*, std::__cxx11::list<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > )

Can anyone advise how to go about it?

Answer 1

Two possible approaches come to mind.

Let's assume we're trying to expose the following C++ class, non-intrusively:

class MyClass
{
public:
    MyClass(std::list<std::string> messages)
        : msgs(std::move(messages))
    {
    }

    void dump() const
    {
        std::cout << "Messages:\n";
        for (auto const& msg : msgs) {
            std::cout << msg << "\n";
        }
    }

    // NB: This would be better returned as const ref
    //     but I have issues exposing that to Python
    //     that I have yet to solve
    std::list<std::string> messages() const
    {
        return msgs;
    }

private:
    std::list<std::string> msgs;
};

---

If the only place we need to deal with std::list is the constructor, then the simplest approach is to write a small "factory" function which will

• Take a Python list as an input.
• Create a std::list and populate it with values from the Python list.
• Instantiate MyClass with the std::list.
• Return that instance of MyClass.

We'll use a shared_ptr to deal with the memory management. To easily initialize the std::list, we can take advantage of boost::python::stl_input_iterator.

boost::shared_ptr<MyClass> create_MyClass(bp::list const& l)
{
    std::list<std::string> messages{ bp::stl_input_iterator<std::string>(l)
        , bp::stl_input_iterator<std::string>() };
    return boost::make_shared<MyClass>(messages);
}

Once we have this function, we'll expose it in place of the original MyClass constructor. To do this, we first need to disable any default constructor bindings, so we use boost::python::no_init. In python, constructors are simply functions named __init__. Finally, we need to use the apparently undocumented function boost::python::make_constructor to create an appriate function object.

BOOST_PYTHON_MODULE(so07)
{
    bp::class_<MyClass, boost::noncopyable, boost::shared_ptr<MyClass>>("MyClass", bp::no_init)
        .def("__init__", bp::make_constructor(create_MyClass))
        .def("dump", &MyClass::dump)
        ;
}

Transcript:

>>> import so07
>>> test = so07.MyClass(['a','b','c'])
>>> test.dump()
Messages:
a
b
c

---

If we wish to use std::list in other contexts, then writing individual wrapper functions to deal with the translation would quickly get out of hand. To avoid this, we can register custom converters that will allow Boost.Python to automatically convert Python lists holding strings to std::list<std::string> objects and vice versa.

Going from C++ to python is quite simple -- just construct a boost::python::list and then add all the elements from the C++ list. We can register this converter using boost::python::to_python_converter.

struct std_list_to_python
{
    static PyObject* convert(std::list<std::string> const& l)
    {
        bp::list result;
        for (auto const& value : l) {
            result.append(value);
        }
        return bp::incref(result.ptr());
    }
};

Going from Python to C++ is a two-step process. First of all, we need a function to determined whether the input is a valid candidate for conversion. In this case, the object needs to be a Python list, and each of its elements needs to be a Python string. The second steps consists of in-place construction of the std::list and subsequent population of it with the elements from the Python list. We register this converter using boost::python::converter::registry::push_back.

struct pylist_converter
{
    static void* convertible(PyObject* object)
    {
        if (!PyList_Check(object)) {
            return nullptr;
        }

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            if (!PyString_Check(PyList_GetItem(object, i))) {
                return nullptr;
            }
        }

        return object;
    }

    static void construct(PyObject* object, bp::converter::rvalue_from_python_stage1_data* data)
    {
        typedef bp::converter::rvalue_from_python_storage<std::list<std::string>> storage_type;
        void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

        data->convertible = new (storage) std::list<std::string>();

        std::list<std::string>* l = (std::list<std::string>*)(storage);

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            l->push_back(bp::extract<std::string>(PyList_GetItem(object, i)));
        }
    }
};

Our module will look as follows:

BOOST_PYTHON_MODULE(so07)
{
    bp::to_python_converter<std::list<std::string>, std_list_to_python>();

    bp::converter::registry::push_back(&pylist_converter::convertible
        , &pylist_converter::construct
        , bp::type_id<std::list<std::string>>());

    bp::class_<MyClass, boost::noncopyable>("MyClass", bp::init<std::list<std::string>>())
        .def("dump", &MyClass::dump)
        .def("messages", &MyClass::messages)
        ;
}

Transcript:

>>> import so07
>>> test = so07.MyClass(['a','b','c'])
>>> test.dump()
Messages:
a
b
c
>>> test.messages()
['a', 'b', 'c']

---

References:

• boost-python: How do I provide a custom constructor wrapper function?
• https://wiki.python.org/moin/boost.python/HowTo#named_constructors_.2F_factories_.28as_Python_initializers.29
• https://www.boost.org/doc/libs/1_70_0/libs/python/doc/html/tutorial/tutorial/exposing.html#tutorial.exposing.constructors
• Feeding a Python list into a function taking in a vector with Boost Python

---

Complete Code:

#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/make_shared.hpp>

#include <list>
#include <iostream>

namespace bp = boost::python;

class MyClass
{
public:
    MyClass(std::list<std::string> messages)
        : msgs(std::move(messages))
    {
    }

    void dump() const
    {
        std::cout << "Messages:\n";
        for (auto const& msg : msgs) {
            std::cout << msg << "\n";
        }
    }

    std::list<std::string> messages() const
    {
        return msgs;
    }

private:
    std::list<std::string> msgs;
};

#if 1

boost::shared_ptr<MyClass> create_MyClass(bp::list const& l)
{
    std::list<std::string> messages{ bp::stl_input_iterator<std::string>(l)
        , bp::stl_input_iterator<std::string>() };
    return boost::make_shared<MyClass>(messages);
}

BOOST_PYTHON_MODULE(so07)
{
    bp::class_<MyClass, boost::noncopyable, boost::shared_ptr<MyClass>>("MyClass", bp::no_init)
        .def("__init__", bp::make_constructor(create_MyClass))
        .def("dump", &MyClass::dump)
        ;
}

#else

struct std_list_to_python
{
    static PyObject* convert(std::list<std::string> const& l)
    {
        bp::list result;
        for (auto const& value : l) {
            result.append(value);
        }
        return bp::incref(result.ptr());
    }
};


struct pylist_converter
{
    static void* convertible(PyObject* object)
    {
        if (!PyList_Check(object)) {
            return nullptr;
        }

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            if (!PyString_Check(PyList_GetItem(object, i))) {
                return nullptr;
            }
        }

        return object;
    }

    static void construct(PyObject* object, bp::converter::rvalue_from_python_stage1_data* data)
    {
        typedef bp::converter::rvalue_from_python_storage<std::list<std::string>> storage_type;
        void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

        data->convertible = new (storage) std::list<std::string>();

        std::list<std::string>* l = (std::list<std::string>*)(storage);

        int sz = PySequence_Size(object);
        for (int i = 0; i < sz; ++i) {
            l->push_back(bp::extract<std::string>(PyList_GetItem(object, i)));
        }
    }
};


BOOST_PYTHON_MODULE(so07)
{
    bp::to_python_converter<std::list<std::string>, std_list_to_python>();

    bp::converter::registry::push_back(&pylist_converter::convertible
        , &pylist_converter::construct
        , bp::type_id<std::list<std::string>>());

    bp::class_<MyClass, boost::noncopyable>("MyClass", bp::init<std::list<std::string>>())
        .def("dump", &MyClass::dump)
        .def("messages", &MyClass::messages)
        ;
}

#endif

Answer 2

It's my first go at this but it turns out boost has their own python list type

This actually does work:

#include <boost/python.hpp>
#include <boost/python/list.hpp>
#include <boost/python/extract.hpp>
#include <string>

using namespace boost::python;

struct MyClass {
    MyClass(boost::python::list messages) : msgs(messages) {}
    void set(boost::python::list messages) { msgs = messages; }
    boost::python::list get() { return msgs; }

    boost::python::list msgs;
};

BOOST_PYTHON_MODULE(my_first) {
    class_<MyClass, boost::noncopyable>("MyClass", init<boost::python::list>())
        .def("get", &MyClass::get)
        .def("set", &MyClass::set);
}

A session:

PYTHONPATH="." python3
>>> from my_first import MyClass
>>> a = MyClass(['a', 'b'])
>>> b = a.get()
>>> print(b)
['a', 'b']