protobuf-net custom serialization and model configuration

Question

So I'm writing a custom serialization system for Unity3D that lets users implement and choose different serializes. I currently support BinaryFormatter and protobuf-net. However; in this system I have persistent custom rules for serialization that I'd like my serializers to play nice with:

1. Types are serializable only if they're annotated with [Serializable]
2. Properties with side effects are not serialized, only the auto ones
3. Public fields/auto-properties are serialized implicitly
4. Non-public fields/auto-properties are serialized only when a custom attribute is applied on them (I have a variety of those: [Save], [Serialize] and [SerializeField]

Now I'd like to adapt my protobuf-net model to adapt to these rules so I don't have to use any of protobufs custom attributes like ProtoContract, ProtoMember etc.

The way I figured I could do this, is by having an array of serializable types that the user would add his custom types to (that way he doesn't need ProtoContract on those types) - I would iterate on those types and add them to my model. Foreach type, I would get the members that would satisfy my serialization rules, and add those to the model too.

Another thing I'd like, is say you have an abstract class A with children B and C, users don't have to explicitly add B and C, they just add A and I would get A's children and add them myself.

My question boils down to: Instead of users having to write this:

[ProtoContract]
[ProtoInclude(1, typeof(Child1))]
[ProtoInclude(2, typeof(Child2))]
public abstract class AbstractBase
{
    public abstract int Num { get; set; }
}

[ProtoContract]
public class Child1 : AbstractBase
{
    [ProtoMember(1)]
    public int x;

    public override int Num { get { return x; } set { x = value; } }
}

[ProtoContract]
public class Child2 : AbstractBase
{
    [ProtoMember(1)]
    public int y;
    [ProtoMember(2)]
    public int z;

    public override int Num { get { return y; } set { y = value; } }
}

I would like them to be able to write this:

[Serializble]
public abstract class AbstractBase
{
    public abstract int Num { get; set; }
}

[Serializble]
public class Child1 : AbstractBase
{
    public int x;

    public override int Num { get { return x; } set { x = value; } }
}

[Serializble]
public class Child2 : AbstractBase
{
    public int y;
    public int z;

    public override int Num { get { return y; } set { y = value; } }
}

// ProtobufSerializableTypes.cs
public static Type[] SerializableTypes = new[]
{
    typeof(AbstractBase)
};

Here's what I tried:

[TestClass]
public class ProtobufDynamicSerializationTestSuite
{
    private AbstractBase Base { get; set; }
    private Type[] SerializableTypes { get; set; }

    [TestInitialize]
    public void Setup()
    {
        Base = new Child1();
        SerializableTypes = new[]
        {
            typeof(AbstractBase)
        };
    }

    [TestMethod]
    public void ShouldCopyWithCustomConfig()
    {
        var model = TypeModel.Create();

        Func<Type, MetaType> addType = type =>
        {
            log("adding type: {0}", type.Name);
            return model.Add(type, false);
        };

        var hierarchy = new Dictionary<MetaType, List<Type>>();
        for (int i = 0; i < SerializableTypes.Length; i++)
        {
            var type = SerializableTypes[i];
            var meta = addType(type);
            var temp = new List<Type>();
            var children = type.Assembly.GetTypes().Where(t => t.IsSubclassOf(type) && !t.IsAbstract).ToList();
            for(int j = 0; j < children.Count; j++)
            {
                var child = children[j];
                addType(child);
                log("adding subtype {0} with id {1}", child.Name, j + 1);
                meta.AddSubType(j + 1, child);
                temp.Add(child);
            }
            hierarchy[meta] = temp;
        }

        Func<Type, string[]> getMemberNames = x =>
            //SerializationLogic.GetSerializableMembers(x, null) // real logic
                x.GetMembers(BindingFlags.Instance | BindingFlags.DeclaredOnly | BindingFlags.Public) // dummy logic
                             .Where(m => m.MemberType == MemberTypes.Field)
                             .Select(m => m.Name)
                             .ToArray();

        foreach (var entry in hierarchy)
        {
            int id = 1;
            foreach (var type in entry.Value)
            {
                foreach (var member in getMemberNames(type))
                {
                    log("adding member {0} to type {1} with id {2}", member, type.Name, id);
                    entry.Key.Add(id++, member);
                }
            }
        }

        Base.Num = 10;
        var copy = (AbstractBase)model.DeepClone(Base);
        Assert.AreEqual(copy.Num, 10);
    }

    void log(string msg, params object[] args)
    {
        Console.WriteLine(string.Format(msg, args));
    }

    void log(string msg)
    {
        log(msg, new object[0]);
    }
}

So my attempt was to add all the necessary types to the model, add all children types to parent types, and then iterate on all added types and add to the model the appropriate fields/properties (that correspond to my serialization rules) from that type

This is failing however to:

Test Name:  ShouldCopyWithCustomConfig
Test Outcome:   Failed
Result Message: 
Test method ProtobufTests.ProtobufDynamicSerializationTestSuite.ShouldCopyWithCustomConfig threw exception: 
System.ArgumentException: Unable to determine member: x
Parameter name: memberName
Result StandardOutput:  
adding type: AbstractBase
adding type: Child1
adding subtype Child1 with id 1
adding type: Child2
adding subtype Child2 with id 2
adding member x to type Child1 with id 1

What am I doing wrong? and is there a better way to do this?

Thanks!

---

Note that initially I didn't have this dictionary step, I tried to add a type's members right after adding it to the model, but this fails if I have say, type A and B, A has a B reference, if I try to add the type A and its members, I will come along B which protobuf is unable to identify at this stage, because it's not added to the model yet... So I thought it was necessary to add the types first, then their members...

Answer 1

The main issue seems to be that entry.Key refers to the base-type, but you are trying to describe members of the specific sub-types; here's what I did:

        foreach (var entry in hierarchy)
        {
            foreach (var type in entry.Value)
            {
                var meta = model.Add(type, false);
                var members = getMemberNames(type);
                log("adding members {0} to type {1}",
                    string.Join(",", members), type.Name);
                meta.Add(getMemberNames(type));
            }
        }

I also added some strict ordering:

.OrderBy(m => m.Name) // in getMemberNames

and

.OrderBy(x => x.FullName) // in var children =

to ensure that the ids are at least predictable. Note that in protobuf, the ids are very important: a consequence of not rigidly defining the ids is that if someone adds AardvarkCount to your model, it could offset all the ids, and break deserialization of existing data. Something to watch out for.