cereal serialization and polymorphism

Question

Ok so I'm running into a problem in C++11 with cereal (http://uscilab.github.io/cereal/).

In an abstract sense I have a large graph which I am serializing with lots of shared pointers connecting the edges and vertices. Edges (and vertices) also have attributes attached to them.

Now one of these attributes (base class) is an account (child class). Account also inherits from Idable which also is serializable. Now here are some pertinent snips of code which show some of my cereal usage. I'll explain the issue after this context:

Attribute.hpp/cpp

class Attribute {
...

template<class Archive> void serialize(Archive&)
{   
}   

friend class cereal::access;
...

CEREAL_REGISTER_TYPE(mgraph::Attribute)

Idable.hpp/cpp

class Idable {
...

Id id;

template<class Archive> void serialize(Archive& archive)
{
    archive(cereal::make_nvp("id", id)); 
}

template<class Archive> static void load_and_construct(Archive& ar, cereal::construct<mcommon::Idable>& construct)
{
    mcommon::Id id;
    ar(id);
    construct(id);
}

friend class cereal::access;
...

CEREAL_REGISTER_TYPE(mcommon::Idable)

Position.hpp/cpp

class Position
: public mgraph::Attribute
  , public mcommon::Displayable {

template<class Archive> void serialize(Archive& archive)
{   
    archive(cereal::make_nvp("Attribute",
                             cereal::base_class<mgraph::Attribute>(this)));
}   

friend class cereal::access;
...

CEREAL_REGISTER_TYPE(mfin::Position)

Account.hpp/cpp

class Account
: public mcommon::Idable
  , public Position {
...
Currency balance;

template<class Archive> void serialize(Archive& archive)
{   
    archive(cereal::make_nvp("Idable",
                             cereal::base_class<mcommon::Idable>(this)),
            cereal::make_nvp("Position",
                             cereal::base_class<mfin::Position>(this)),
            cereal::make_nvp("balance", balance));
}

template<class Archive> static void load_and_construct(Archive& ar, cereal::construct<Account>& construct)
{
    mcommon::Id iden;
    Currency::Code code;
    ar(iden, code);
    construct(iden, code);
}

friend class cereal::access;
...

CEREAL_REGISTER_TYPE(mfin::Account)

So the problem comes when a mfin::Account is being serialized. The mfin::Account belongs to a std::list>. When we get down into the serialize function for Idable the object is invalid.

Going into gdb which halts on a segfault I go up a few stackframes to this this line: /usr/include/cereal/types/polymorphic.hpp:341. Which is:

(gdb) list
336 
337     auto binding = bindingMap.find(std::type_index(ptrinfo));
338     if(binding == bindingMap.end())
339       UNREGISTERED_POLYMORPHIC_EXCEPTION(save, cereal::util::demangle(ptrinfo.name()))
340 
341     binding->second.shared_ptr(&ar, ptr.get());
342   }
343 
344   //! Loading std::shared_ptr for polymorphic types
345   template <class Archive, class T> inline

Now here this is what ptr is:

(gdb) print *((mfin::Account*)(ptr.get()))
$10 = {<mcommon::Idable> = {_vptr.Idable = 0x4f0d50 <vtable for mfin::Account+16>, id = "bank"}, <mfin::Position> = {<mgraph::Attribute> = {
      _vptr.Attribute = 0x4f0d78 <vtable for mfin::Account+56>}, <mcommon::Displayable> = {_vptr.Displayable = 0x4f0da0 <vtable for mfin::Account+96>}, <No data fields>}, balance = {<mcommon::Displayable> = {
      _vptr.Displayable = 0x4f0570 <vtable for mfin::Currency+16>}, amount = 0, code = mfin::Currency::USD}}
(gdb) print ptr
$11 = std::shared_ptr (count 3, weak 0) 0x758ad0

Everything is looking good. But notice when I cast it to a void*:

$11 = std::shared_ptr (count 3, weak 0) 0x758ad0
(gdb) print *((mfin::Account*)((void*)ptr.get()))
$12 = {<mcommon::Idable> = {_vptr.Idable = 0x4f0d78 <vtable for mfin::Account+56>, 
    id = "\363aL\000\000\000\000\000PbL\000\000\000\000\000\304\031L\000\000\000\000\000\021#L", '\000' <repeats 13 times>, " \232N", '\000' <repeats 21 times>, "P\251@\000\000\000\000\000\370\377\377\377\377\377\377\377 \232N", '\000' <repeats 21 times>, "\304\031L\000\000\000\000\000P\251@", '\000' <repeats 45 times>, "St19_Sp_counted_deleterIPN4mfin7AccountE"...}, <mfin::Position> = {<mgraph::Attribute> = {
      _vptr.Attribute = 0x4f0570 <vtable for mfin::Currency+16>}, <mcommon::Displayable> = {_vptr.Displayable = 0x0}, <No data fields>}, balance = {<mcommon::Displayable> = {_vptr.Displayable = 0x0}, amount = 49, 
    code = (unknown: 7702648)}}

This is of course what happens in binding->second.shared_ptr (seen below) which takes a const void*.

(gdb) list
295             writeMetadata(ar);
296 
297             #ifdef _MSC_VER
298             savePolymorphicSharedPtr( ar, dptr, ::cereal::traits::has_shared_from_this<T>::type() ); // MSVC doesn't like typename here
299             #else // not _MSC_VER
300             savePolymorphicSharedPtr( ar, dptr, typename ::cereal::traits::has_shared_from_this<T>::type() );
301             #endif // _MSC_VER
302           };
303 
304         serializers.unique_ptr =

What is wrong in my usage of cereal that would cause this? Here is the final error I get:

Program received signal SIGSEGV, Segmentation fault.
0x000000000040f7cd in rapidjson::Writer<rapidjson::GenericWriteStream, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::WriteString (this=0x7fffffffd358, 
    str=0x4f1ae0 <vtable for mfin::Account+96> "\363aL", length=4989722) at /usr/include/cereal/external/rapidjson/writer.h:276
276             if ((sizeof(Ch) == 1 || characterOk(*p)) && escape[(unsigned char)*p])  {
Missing separate debuginfos, use: debuginfo-install boost-date-time-1.55.0-8.fc21.x86_64 boost-filesystem-1.55.0-8.fc21.x86_64 boost-program-options-1.55.0-8.fc21.x86_64 boost-system-1.55.0-8.fc21.x86_64 boost-thread-1.55.0-8.fc21.x86_64 fcgi-2.4.0-24.fc21.x86_64 glog-0.3.3-3.128tech.x86_64 libgcc-4.9.2-1.fc21.x86_64 libstdc++-4.9.2-1.fc21.x86_64

Answer 1

This isn't an answer for your specific question, but it's very similar. I was trying to serialize a struct with an external serialize function but I was getting an error about my derived struct not being convertible to the base struct type. The trick was that I wasn't passing a pointer.

So

ar(cereal::base_class<Base>(derived_instance), derived_instance.y)

became

ar(cereal::base_class<Base>(&derived_instance), derived_instance.y)

any it compiles fine!

Answer 2

I think I had a similar problem. Some statics failed to initialize properly in cereal. Finally, I was able to find a solution, by placing this code at the source file that instantiates input archives.

#define SC_REGISTER_INPUT_ARCHIVE(Archive) \
namespace cereal    \
{   \
    namespace detail    \
    {   \
        template StaticObject<InputBindingCreator<Archive, first_polymorphic_class>>;   \
        template StaticObject<InputBindingCreator<Archive, second_polymorphic_class>>;  \
... /* repeat for all polymorphic serialized types */
    }   \
}

SC_REGISTER_INPUT_ARCHIVE(XMLInputArchive);
SC_REGISTER_INPUT_ARCHIVE(BinaryInputArchive);

Answer 3

Ok after much investigation I believe I have the answer to my problem. And I believe this is a bug in the library. After I have confirmed this with the library owners I will ensure this is up to date with the results there.

I have produced a simple program below which demonstrates this problem. The issue stems from multiple inheritance, polymorphism, and casting. In the program below notice were we create a Derived object. The Derived object when laid out in memory will have a format approximately.:

Derived:
  Base2::vtable
  Base2::var
  Base::vtable

Consider:

(gdb) print ptr
$2 = std::shared_ptr (count 1, weak 0) 0x63c580
(gdb) print *ptr
$3 = (Derived &) @0x63c580: {<Base2> = {_vptr.Base2 = 0x421f90 <vtable for Derived+16>, var = ""}, <Base> = {_vptr.Base = 0x421fa8 <vtable for Derived+40>}, <No data fields>}

Now when we dynamic_pointer_cast it to Base we have:

(gdb) print ptr
$8 = std::shared_ptr (count 2, weak 0) 0x63c590
(gdb) print *ptr
$9 = (Base &) @0x63c590: {_vptr.Base = 0x421fa8 <vtable for Derived+40>}

This is where the problem begins. Now on /usr/include/cereal/types/polymorphic.hpp, line 341. We have this ptr to Base. Here we have:

binding->second.shared_ptr(&ar, ptr.get());

Which ends up being a cast to a const void*. Later on based on the type info however we cast this the type from the registered polymorphic type. Since the shared_ptr points to an object of the Derived type this means a Derived*. As seen below:

272       static inline void savePolymorphicSharedPtr( Archive & ar, void const * dptr, std::false_type /* has_shared_from_this */ )
273       {
274         PolymorphicSharedPointerWrapper psptr( dptr );
275         ar( CEREAL_NVP_("ptr_wrapper", memory_detail::make_ptr_wrapper( psptr() ) ) );
276       }

Now this means down the stack ptr which is a Base* was cast to void* and then cast to Derived*. And thus the cast chain results in an invalid object. As seen below this the ptr is invalid now:

(gdb) print *ptr
$7 = (const Derived &) @0x63c590: {<Base2> = {_vptr.Base2 = 0x421fa8 <vtable for Derived+40>, var = <error reading variable: Cannot access memory at address 0x49>}, <Base> = {_vptr.Base = 0x0}, <No data fields>}

The pointer is pointing to the vtable for Base and not Derived/Base2 like it should be thus the program crashes:

{
    "ptr": {
        "polymorphic_id": 2147483649,
        "polymorphic_name": "Derived",
        "ptr_wrapper": {
            "id": 2147483649,
            "data": {
                "Base2": {

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b8e9e3 in std::string::size() const () from /lib64/libstdc++.so.6

Below is a sample program which reproduces this:

// g++ test.cpp -std=c++11 -ggdb -o test && gdb ./test
#include <cereal/archives/json.hpp>
#include <cereal/types/polymorphic.hpp>
#include <iostream>

struct Base {
    virtual void foo() { } 
    template<class Archive> void serialize(Archive& archive) { } 
};

struct Base2 {
    virtual void foo() { } 
    std::string var;
    template<class Archive> void serialize(Archive& archive) {   
        archive(cereal::make_nvp("var", var));
    }   
};

struct Derived : public Base2, public Base {
    template<class Archive> void serialize(Archive& archive) {   
        archive(cereal::make_nvp("Base2",
                                 cereal::base_class<Base2>(this)),
                cereal::make_nvp("Base",
                                 cereal::base_class<Base>(this)));
    }   
};

CEREAL_REGISTER_TYPE(Base);
CEREAL_REGISTER_TYPE(Base2);
CEREAL_REGISTER_TYPE(Derived);

int main() {
    auto ptr = std::make_shared<Derived>();
    cereal::JSONOutputArchive ar(std::cout);
    ar(cereal::make_nvp("ptr", std::dynamic_pointer_cast<Base>(ptr)));

    return 0;
}