Data Contract Serializer Forward Compatibility of properties which are referenced

Question

I am trying to support Forward Compatibility for Data Contract Serializer. The case which I have problems with:

If you have an object which is saved as a reference to a property which is added in a later version inside of a known type, it will become an exception. Note that both of these types are known in both versions. The only thing which is new is the property inside of one of the objects.

I attached a simple simulation of the problem at samples and samples:

It has two different projects: V1 which is an older version which is already deployed. V2 which is a newer version of V1. V2 is saving its data and V1 needs to be able to load V2 saved data to support forward compatibility.

There are three custom types: People: has two object references and Person and AnotherPerson are being saved in them.

In V1 and V2:

[DataContract(Name = "People", Namespace = "Tests.FCTests")]
[KnownType(typeof(Person))]
[KnownType(typeof(AnotherPerson))]
public class People : IExtensibleDataObject
{
    [DataMember]
    public object Person { get; set; }

    [DataMember]
    public object AnotherPerson { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }
}

Person: has a Name.

In V1 and V2:

[DataContract(Name = "Person", Namespace = "Tests.FCTests")]
public class Person : IExtensibleDataObject
{
    [DataMember]
    public string Name { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }

}

AnotherPerson: has a Name and in V2 a reference to Person (FriendPerson) was added.

In V1:

[DataContract(Name = "AnotherPerson", Namespace = "Tests.FCTests")]
public class AnotherPerson : IExtensibleDataObject
{
    [DataMember]
    public string Name { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }
}

In V2:

[DataContract(Name = "AnotherPerson", Namespace = "Tests.FCTests")]
public class AnotherPerson : IExtensibleDataObject
{
    [DataMember]
    public string Name { get; set; }

    /* This is added in this version */
    [DataMember]
    public Person FriendPerson { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }
}

Version 2 is saving the data:

    static void Main(string[] args)
    {
        DataContractSerializer serializer = new DataContractSerializer(typeof(People), null, int.MaxValue, false, true, null, null);

        var people = new People();
        var person = new Person() { Name = "Person" };
        var anotherPerson = new AnotherPerson() { Name = "AnotherPerson", FriendPerson = person };

        people.Person = person;
        people.AnotherPerson = anotherPerson;

        using (var writer = new XmlTextWriter("../../../../SavedFiles/Version2Saved.xml", null) { Formatting = Formatting.Indented })
        {
            serializer.WriteObject(writer, people);
            writer.Flush();
        }

        Console.WriteLine("Save Successfull.");
        Console.ReadKey();
    }

Version 1 is loading the same data:

    static void Main(string[] args)
    {
        DataContractSerializer serializer = new DataContractSerializer(typeof(People), null, int.MaxValue, false, true, null, null);

        People loadedPeople;

        using (var reader = new XmlTextReader("../../../../SavedFiles/Version2Saved.xml"))
        {
            loadedPeople = (People)serializer.ReadObject(reader);
        }

        Console.WriteLine("Load Successful.");

        Console.ReadKey();
    }

The saved data:

<People xmlns:i="http://www.w3.org/2001/XMLSchema-instance" z:Id="1" xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/" xmlns="Tests.FCTests">
  <AnotherPerson z:Id="2" i:type="AnotherPerson">
    <FriendPerson z:Id="3">
      <Name z:Id="4">Person</Name>
    </FriendPerson>
    <Name z:Id="5">AnotherPerson</Name>
  </AnotherPerson>
  <Person z:Ref="3" i:nil="true" />
</People>

When V1 is trying to load the data this exception is thrown:

{System.Runtime.Serialization.SerializationException: Element Person from namespace Tests.FCTests cannot have child contents to be deserialized as an object. Please use XmlNode[] to deserialize this pattern of XML. ---> System.Xml.XmlException: 'Element' is an invalid XmlNodeType.
   at System.Xml.XmlReader.ReadEndElement()
   at System.Runtime.Serialization.XmlReaderDelegator.ReadEndElement()
   at System.Runtime.Serialization.ObjectDataContract.ReadXmlValue(XmlReaderDelegator reader, XmlObjectSerializerReadContext context)
   --- End of inner exception stack trace ---
   at System.Runtime.Serialization.ObjectDataContract.ReadXmlValue(XmlReaderDelegator reader, XmlObjectSerializerReadContext context)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, String name, String ns)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.DeserializeFromExtensionData(IDataNode dataNode, Type type, String name, String ns)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.GetExistingObject(String id, Type type, String name, String ns)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.TryHandleNullOrRef(XmlReaderDelegator reader, Type declaredType, String name, String ns, Object& retObj)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
   at ReadPeopleFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
   at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
   at System.Runtime.Serialization.DataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.DataContractSerializer.ReadObject(XmlReader reader)
   at Version1.Program.Main(String[] args) in C:\Users\Administrator\Desktop\Unknown types Test\Version1\Version1\Program.cs:line 17}

Inner Exception:

{System.Xml.XmlException: 'Element' is an invalid XmlNodeType.
   at System.Xml.XmlReader.ReadEndElement()
   at System.Runtime.Serialization.XmlReaderDelegator.ReadEndElement()
   at System.Runtime.Serialization.ObjectDataContract.ReadXmlValue(XmlReaderDelegator reader, XmlObjectSerializerReadContext context)}

I suspect that the error is because the object is referencing to a type which is being deserialized inside of Extension Object and does not have any type. The reason is, if you add a new instance of Person inside of People and not referencing the same instance inside of AnotherPerson (FriendPerson).

var anotherPerson = new AnotherPerson() { Name = "AnotherPerson", FriendPerson = new Person() };

Then the saved file becomes the following and everything works just fine:

<People xmlns:i="http://www.w3.org/2001/XMLSchema-instance" z:Id="1" xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/" xmlns="Tests.FCTests">
  <AnotherPerson z:Id="2" i:type="AnotherPerson">
    <FriendPerson z:Id="3">
      <Name i:nil="true" />
    </FriendPerson>
    <Name z:Id="4">AnotherPerson</Name>
  </AnotherPerson>
  <Person z:Id="5" i:type="Person">
    <Name z:Id="6">Person</Name>
  </Person>
</People>

I tried to fix the problem using a Data Contract Resolver, Adding Known Types inside of serializer dynamically, and Data Contract Surrogate but, none of them worked. The reason is the exception is thrown when the serializer is de-serializing FriendPerson and the overridden methods inside of the surrogate or resolver are not being called prior to that.

NOTE We need to preserve object references and removing it is not an option.

Answer 1

I was communicating with MSDN incident support and after 2 months of going back and forth, they answered:

We have engaged product group and the official word is there is a bug in IExtensibleDataObject (when cyclic references are ON).

I hope they add this somewhere in their documentation and I hope this helps future developments for the others.

Answer 2

The problem is the order of the fields in the V2 of the Person data-contract. The new field needs to be appended at the end of the serialized document in order to be forward-compatible:

<People xmlns:i="http://www.w3.org/2001/XMLSchema-instance" z:Id="1" xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/" xmlns="Tests.FCTests">
  <AnotherPerson z:Id="2" i:type="AnotherPerson">
    <FriendPerson z:Id="3">
      <Name z:Id="4">Person</Name>
    </FriendPerson>
    <Name z:Id="5">AnotherPerson</Name>
  </AnotherPerson>
  <Person z:Ref="3" i:nil="true" />
</People>

Note how the "FriendPerson" tag in the above XML appears above the "Name" tag in the "AnotherPerson" segment. It would work if your object had been serialized as follows:

<People xmlns:i="http://www.w3.org/2001/XMLSchema-instance" z:Id="1" xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/" xmlns="Tests.FCTests">
  <AnotherPerson z:Id="2" i:type="AnotherPerson">
    <Name z:Id="5">AnotherPerson</Name>
    <FriendPerson z:Id="3">
      <Name z:Id="4">Person</Name>
    </FriendPerson>
  </AnotherPerson>
  <Person z:Ref="3" i:nil="true" />
</People>

To achieve this, specify the "Order" parameter on the DataMemberAttribute of "FriendPerson" property of the V2 "AnotherPerson" class as follows:

[DataContract(Name = "AnotherPerson", Namespace = "Tests.FCTests")]
public class AnotherPerson : IExtensibleDataObject
{
    [DataMember]
    public string Name { get; set; }

    /* This is added in this version */
    [DataMember(Order = 2)]
    public Person FriendPerson { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }
}

As a general rule, you should not use the "Order" parameter in the first version of your datacontract. For any newer version, you should specify the "Order" parameter on any new DataMemberAttribute and increment the specified number along with the version number. It is perfectly legal to have several identical "Order" parameter values in a single datacontract such as in this V3:

[DataContract(Name = "AnotherPerson", Namespace = "Tests.FCTests")]
public class AnotherPerson : IExtensibleDataObject
{
    [DataMember]
    public string Name { get; set; }

    /* This is added in this version */
    [DataMember(Order = 2)]
    public Person FriendPerson { get; set; }

    [DataMember(Order = 3)]
    public string Remarks { get; set; }

    [DataMember(Order = 3)]
    public bool? IsMarried { get; set; }

    public ExtensionDataObject ExtensionData { get; set; }
}

P.S.: My answer may come late, but may still be helpful to others...