Converting a std::list of structs to a Python list

Question

I'm working on an accelerator module for Python written in C++. It's an implementation of Dijkstra's algorithm and eventually returns a pointer to a Route object which in turn contains a std::list<struct waypoint>. To interface with this object using Python I wrote the following interface file:

%{
#include "universe.hpp"
%}

%include <std_list.i>
%include "universe.hpp"

%template(Waypoints) std::list<struct waypoint>;

This allowed me to access the list object and index it, but the iterator over it was not working properly:

>>> r
<mod.Route; proxy of <Swig Object of type 'Route *' at 0x7f6ab39ff4e0> >
>>> r.points
<mod.Waypoints; proxy of <Swig Object of type 'std::list< waypoint > *' at 0x7f6ab502d630> >
>>> r.points[0]
<mod.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7f6ab39ff540> >
>>> for x in r.points:
...     print(x)
... 
<Swig Object of type 'unknown' at 0x7f6ab39ff5d0>
swig/python detected a memory leak of type 'unknown', no destructor found.
<Swig Object of type 'unknown' at 0x7f6ab39ff5a0>
swig/python detected a memory leak of type 'unknown', no destructor found.
<Swig Object of type 'unknown' at 0x7f6ab39ff5d0>
swig/python detected a memory leak of type 'unknown', no destructor found.
...

In an attempt to remedy this (and to make the code bit nicer to read) I wanted to convert the std::list<struct waypoint> into a normal Python list so I wrote the following typemap:

%{
#include "universe.hpp"
%}

%typemap(out) std::list<struct waypoint> {
    $result = PyList_New($1->size());

    printf("This never gets printed\n");

    for(size_t i = 0; i < $1->size(); ++i) {
        PyList_SET_ITEM($result, i, $1[i]);
    }
}

%include <std_list.i>

%include "universe.hpp"

This typemap doesn't work, however. In fact, the list is now just a SwigPyObject and does not support subscription at all. The typemap is also never executed since the print statement inside it is never executed. I'm having a hard time finding people with the same problem on the web and I hope someone here can help me figure out what I'm doing wrong.

These are the minimal contents of universe.hpp:

#include <string>
#include <list>

struct Entity;

struct waypoint {
    Entity *entity;
    int type;
};

class Route {
public:
    int loops;
    double cost;
    std::list<struct waypoint> points;
};

class Entity {
public:
    float x, y, z;
    int id, seq_id;
    std::string name;
};

Answer 1

This initially looked like a SWIG bug to me. It may well not be, and the simplest workaround is trivial change the line:

%template(Waypoints) std::list<struct waypoint>;

To simply be:

%template(Waypoints) std::list<waypoint>;

And all your troubles will be over. This is sufficient to allow the following code to run successfully:

import test
r=test.Route()
r.points[0]

for x in r.points:
    print(x)

Which prints only:

<test.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7fe439e22ed0> >
<test.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7fe439ea5360> >

---

In the interest of completeness here is the journey I took to realise that:

Essentially it seems that there's not a specialization of swig_type_info *swig::type_info<waypoint>() being generated, so the SwigPyIterator::next() call, via SwigPyIterator::value() ends up with a null pointer for swig_type_info for your waypoint type when it calls SWIG_InternalNewPointerObj to generate the Python proxy object. That's causing the unknown type information you see later.

In the meantime though there are two ways to side step that. Firsly in Python you can still trivially get a Python list from a C++ std::list using something like:

list(map(r.points.__getitem__, range(r.points.size())))

Secondly, we can add our own specialization, it's fairly simple to do:

%{
#include "universe.hpp"

  // Workaround: add missing specialization
  namespace swig {
    // Forward declare first
    template <typename T>
    swig_type_info *type_info();

    template <>
    swig_type_info *type_info<waypoint>() {
      return SWIGTYPE_p_waypoint;
    };
  }

%}

%include <std_list.i>
%include "universe.hpp"

%template(Waypoints) std::list<struct waypoint>;

If we dig around a little further in a debugger though we see that the default implementation of type_info() actually calls traits_info<Type>::type_info(). That in turn calls type_query(). And break pointing here we can see what the problem is more clearly - the query we've build has the word "struct " prepended to it. And the swig_type_info struct generated for waypoint doesn't have the word struct anywhere, so the type query system fails.

I am somewhat surprised that the type query function does this with strings, even when the type is known at compile time - the specialisation I used above is far simpler and could seemingly be generated easily. But that's not actually why it didn't work here and the real solution is to drop the word struct from the %template invocation.