Eigen Matrix + Boost::Serialization / C++17

Question

I'm trying to enable C++17 for our code base which is strongly based on boost - and boost::serialization for intermediate data storage and pre-transmission serialization.

Overall, everything looks fine and seems to be working, except when we're serializing Eigen::Matrix objects and include the boost serialization support header for shared ptr serialization.

Minimal example/testcode on github: https://github.com/nightsparc/EigenSerialize

[EDIT] @Marc Glisse provided a reduced testcase down below. See: https://stackoverflow.com/a/54536756/1267320

I did some tests with different compilers (GCC6/7/8 and Clang6). We're normally using the system GCC which is GCC7.3 for Ubuntu 18.04. For me, it seems to be a problem related with the C++17 mode of GCC7 and higher.

I mean, I'm not using a shared_ptr in the minimal example, so I could remove it and everything would be fine...nevertheless, in our codebase shared_ptrs get serialized everywhere.

Does one of you guys have any idea whats going on here? Or is it a bug in GCC C++17 mode?

The test code (without proper error handling and stuff...):

#include <fstream>

#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/serialization/split_free.hpp>

#include <Eigen/Core>

// !! Conflicting include! Whenever the serialization wrapper for shared_ptrs is included
// the compilation fails!
// /usr/local/include/Eigen/src/Core/util/ForwardDeclarations.h:32:
// error: incomplete type ‘Eigen::internal::traits<boost::serialization::U>’ used in nested name specifier
// enum { has_direct_access = (traits<Derived>::Flags & DirectAccessBit) ? 1 : 0,
#include <boost/serialization/shared_ptr.hpp>

// Serialization methods for fixed-size Eigen::Matrix type
namespace boost {
    namespace serialization {
        template<
                class Archive,
                typename _Scalar,
                int _Rows,
                int _Cols,
                int _Options,
                int _MaxRows,
                int _MaxCols
                >
        inline void serialize(Archive & arArchive,
                              Eigen::Matrix<_Scalar,
                              _Rows,
                              _Cols,
                              _Options,
                              _MaxRows,
                              _MaxCols> & arMatrix,
                              const unsigned int aVersion)
        {
            boost::serialization::split_free(arArchive, arMatrix, aVersion);
        }

        template<
                class Archive,
                typename _Scalar,
                int _Rows,
                int _Cols,
                int _Options,
                int _MaxRows,
                int _MaxCols
                >
        inline void save(Archive & arArchive,
                         const Eigen::Matrix<_Scalar,
                         _Rows,
                         _Cols,
                         _Options,
                         _MaxRows,
                         _MaxCols> & arMatrix,
                         const unsigned int)
        {
            typedef typename Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Index TEigenIndex;

            const TEigenIndex lRows = arMatrix.rows();
            const TEigenIndex lCols = arMatrix.cols();

            arArchive << lRows;
            arArchive << lCols;

            if(lRows > 0 && lCols > 0)
            {
                arArchive & boost::serialization::make_array(arMatrix.data(), arMatrix.size());
            }
        }

        template<
                class Archive,
                typename _Scalar,
                int _Rows,
                int _Cols,
                int _Options,
                int _MaxRows,
                int _MaxCols
                >
        inline void load(Archive & arArchive,
                         Eigen::Matrix<_Scalar,
                         _Rows,
                         _Cols,
                         _Options,
                         _MaxRows,
                         _MaxCols> & arMatrix,
                         const unsigned int)
        {
            typedef typename Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Index TEigenIndex;

            TEigenIndex lRows, lCols;

            // deserialize meta data
            arArchive & lRows;
            arArchive & lCols;

            // do some error handling here

            if(lRows > 0 && lCols > 0)
            {
                // deserialize data
                arArchive & boost::serialization::make_array(arMatrix.data(), arMatrix.size());
            }
        }

    }
}

class TestClass
{
    public:
        TestClass()
        {
            // fill eigen
            m(0,0) = 3;
            m(1,0) = 2.5;
            m(0,1) = -1;
            m(1,1) = m(1,0) + m(0,1);
        }

    private:
        friend class boost::serialization::access;
        Eigen::Matrix2d m;

        template<class Archive>
        void serialize(Archive &ar, const unsigned int)
        {
            ar & m;
        }
};


int main(void)
{
    using namespace boost::archive;

    // Serialize
    TestClass TestA;
    std::ofstream oss("test.log");
    {
        text_oarchive oa(oss);
        oa << TestA;
    }

    // deserialize now
    TestClass TestB;
    std::ifstream iss("test.log");
    {
        text_iarchive ia(iss);
        ia >> TestB;
    }
}

[EDIT 2019-02-06]
GCC-Bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84075
Eigen-Bug: http://eigen.tuxfamily.org/bz/show_bug.cgi?id=1676

[EDIT 2019-02-07]
Boost PR: https://github.com/boostorg/serialization/pull/144

Answer 1

Not sure about the error, but why do you need the boost::serialization::split_free instead of simply doing this:

// Serialization methods for fixed or dynamic-size Eigen::Matrix type
namespace boost {namespace serialization {
template<class Archive, typename _Scalar, int _Rows, int _Cols, int _Options, int _MaxRows, int _MaxCols>
inline void serialize(Archive & ar,
        Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols> & matrix,
        const unsigned int /* aVersion */)
{
    Eigen::Index rows = matrix.rows();
    Eigen::Index cols = matrix.cols();
    ar & (rows);
    ar & (cols);
    if(rows != matrix.rows() || cols != matrix.cols())
        matrix.resize(rows, cols);
    if(matrix.size() !=0)
        ar &  boost::serialization::make_array(matrix.data(), rows * cols);
}
} } // namespace boost::serialization

Works fine for me with C++17 with boost 1.58 and the most recent Eigen3.3 or Eigen-default versions on clang 5/6, and gcc 6/7/8.

I added a matrix.resize() which should make the code work for dynamic matrices as well, for fixed sized matrices this should introduce no overhead (when compiled with optimization) -- actually it should assert when reading a non-resizable matrix (when compiled without -DNDEBUG).

---

If you want to keep your current split_free-based serialization you can work-around the issue by adding this template specialization. It needs to be somewhere after including Eigen/Core, before declaring your serialization, it doesn't matter if <boost/serialization/shared_ptr.hpp> is included or not.

namespace boost { namespace serialization {
   struct U;  // forward-declaration for Bug 1676
} } // boost::serialization 

namespace Eigen { namespace internal {
  // Workaround for bug 1676
  template<>
  struct traits<boost::serialization::U> {enum {Flags=0};};
} } 

Answer 2

I would guess (not reliably) that the difference between C++14 and C++17 is due to template argument deduction (the compiler cannot dismiss a template just because the number of parameters is too small), that gcc does not handle as SFINAE. I don't know if the bug is in gcc or Eigen, but here is a more reduced testcase anyway

#include <Eigen/Core>

template<template<class U>class SPT>void f(SPT<class U>&);
template<typename _Scalar, int _Rows, int _Cols, int _Options, int _MaxRows, int _MaxCols>
void f(Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols> & arMatrix){}

int main()
{
  Eigen::Matrix2d m;
  f(m);
}