Json.net - How to preserve dictionary value references when populating a dictionary?

Question

I would like to populate the objects contained within a Dictionary from a JSON file while preserving the object references themselves.

Json.net documentation on PreserveReferencesHandling clearly state that it will not work in case a type implements System.Runtime.Serialization.ISerializable:

Specifies reference handling options for the Newtonsoft.Json.JsonSerializer. Note that references cannot be preserved when a value is set via a non-default constructor such as types that implement System.Runtime.Serialization.ISerializable.

Here is my failing code:

class Model
{
   public int Val { get; set; } = 123;
}

...

    var model = new Model();
    var to_serialize = new Dictionary<int, Model> { { 0, model } }; // works ok with list<Model>

    // serialize
    var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);

    var jsonSerializerSettings = new JsonSerializerSettings();
    jsonSerializerSettings.MissingMemberHandling = MissingMemberHandling.Ignore;
    jsonSerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.All; // does not work for ISerializable
    
    Assert.AreSame(to_serialize[0], model); // ok!

    JsonConvert.PopulateObject(
        value: jsonString,
        target: to_serialize,
        settings: jsonSerializerSettings
    );

    Assert.AreSame(to_serialize[0], model); // not ok... works ok with list<Model>

My main requirement is that when calling PopulateObject(), the constructor of the Model class will not be invoked. Instead, only its internal field will be updated with the value from the JSON. In my real case, the Model class contains other values which are not in the JSON and which I don't want to lose:

[JsonObject(MemberSerialization.OptIn)]
class Model
{
   [JsonProperty(PropertyName = "val_prop")]
   public int Val { get; set; } = 123;

   // not in the json file, would like this field to maintain the value
   // it had prior to PopulateObject()
   public int OtherVal { get; set; } = 456;
}

Is there a way to make this work?

Answer 1

Your problem is similar to the one from JsonSerializer.CreateDefault().Populate(..) resets my values: you would like to populate a preexisting collection, specifically a Dictionary<int, T> for some T, and populate the preexisting values. Unfortunately, in the case of a dictionary, Json.NET will replace the values rather than populate them, as can be seen in JsonSerializerInternalReader.PopulateDictionary() which simply deserializes the value to the appropriate type, and sets it the dictionary.

To work around this limitation, you can create a custom JsonConverter for Dictionary<TKey, TValue> when TKey is a primitive type and TValue is a complex type which merges the incoming JSON key/value pairs onto the preexisting dictionary. The following converter does the trick:

public class DictionaryMergeConverter : JsonConverter
{
    static readonly IContractResolver defaultResolver = JsonSerializer.CreateDefault().ContractResolver;
    readonly IContractResolver resolver = defaultResolver;

    public override bool CanConvert(Type objectType)
    {
        var keyValueTypes = objectType.GetDictionaryKeyValueType();
        if (keyValueTypes == null)
            return false;
        var keyContract = resolver.ResolveContract(keyValueTypes[0]);
        if (!(keyContract is JsonPrimitiveContract))
            return false;
        var contract = resolver.ResolveContract(keyValueTypes[1]);
        return contract is JsonContainerContract;
        // Also possibly check whether keyValueTypes[1] is a read-only collection or dictionary.
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
            return null;
        if (reader.TokenType != JsonToken.StartObject)
            throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
        IDictionary dictionary = existingValue as IDictionary ?? (IDictionary)serializer.ContractResolver.ResolveContract(objectType).DefaultCreator();
        var keyValueTypes = objectType.GetDictionaryKeyValueType();
        while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndObject)
        {
            switch (reader.TokenType)
            {
                case JsonToken.PropertyName:
                    var name = (string)reader.Value;
                    reader.ReadToContentAndAssert();

                    // TODO: DateTime keys and enums with overridden names.
                    var key = (keyValueTypes[0] == typeof(string) ? (object)name : Convert.ChangeType(name, keyValueTypes[0], serializer.Culture));
                    var value = dictionary.Contains(key) ? dictionary[key] : null;

                    // TODO:
                    //  - JsonConverter active for valueType, either in contract or in serializer.Converters
                    //  - NullValueHandling, ObjectCreationHandling, PreserveReferencesHandling, 

                    if (value == null)
                    {
                        value = serializer.Deserialize(reader, keyValueTypes[1]);
                    }
                    else
                    {
                        serializer.Populate(reader, value);
                    }
                    dictionary[key] = value;
                    break;

                default:
                    throw new JsonSerializationException(string.Format("Unexpected token {0}", reader.TokenType));
            }
        }

        return dictionary;
    }

    public override bool CanWrite { get { return false; } }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) { throw new NotImplementedException(); }
}

public static partial class JsonExtensions
{
    public static JsonReader ReadToContentAndAssert(this JsonReader reader)
    {
        return reader.ReadAndAssert().MoveToContentAndAssert();
    }

    public static JsonReader MoveToContentAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (reader.TokenType == JsonToken.None)       // Skip past beginning of stream.
            reader.ReadAndAssert();
        while (reader.TokenType == JsonToken.Comment) // Skip past comments.
            reader.ReadAndAssert();
        return reader;
    }

    public static JsonReader ReadAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (!reader.Read())
            throw new JsonReaderException("Unexpected end of JSON stream.");
        return reader;
    }
}

public static class TypeExtensions
{
    public static IEnumerable<Type> BaseTypesAndSelf(this Type type)
    {
        while (type != null)
        {
            yield return type;
            type = type.BaseType;
        }
    }

    public static Type[] GetDictionaryKeyValueType(this Type type)
    {
        return type.BaseTypesAndSelf().Where(t => t.IsGenericType && t.GetGenericTypeDefinition() == typeof(Dictionary<,>)).Select(t => t.GetGenericArguments()).FirstOrDefault();
    }
}

Having done so, you will encounter a secondary issue: Json.NET will never use a custom converter to populate the root object. To work around this you will need to call JsonConverter.ReadJson() directly, from some utility method:

public static partial class JsonExtensions
{
    public static void PopulateObjectWithConverter(string value, object target, JsonSerializerSettings settings)
    {
        if (target == null || value == null)
            throw new ArgumentNullException();
        var serializer = JsonSerializer.CreateDefault(settings);
        var converter = serializer.Converters.Where(c => c.CanConvert(target.GetType()) && c.CanRead).FirstOrDefault() ?? serializer.ContractResolver.ResolveContract(target.GetType()).Converter;
        using (var jsonReader = new JsonTextReader(new StringReader(value)))
        {
            if (converter == null)
                serializer.Populate(jsonReader, target);
            else
            {
                jsonReader.MoveToContentAndAssert();
                var newtarget = converter.ReadJson(jsonReader, target.GetType(), target, serializer);
                if (newtarget != target)
                    throw new JsonException(string.Format("Converter {0} allocated a new object rather than populating the existing object {1}.", converter, value));
            }
        }
    }
}

You will now be able to populate your dictionary as follows:

var jsonString = JsonConvert.SerializeObject(to_serialize, Formatting.Indented);

var settings = new JsonSerializerSettings
{
    Converters = { new DictionaryMergeConverter() },
};
JsonExtensions.PopulateObjectWithConverter(jsonString, to_serialize, settings);

Notes:

• PreserveReferencesHandling has no impact on whether dictionary values are populated or replaced. Instead this setting controls whether a serialization graph with multiple references to the same object will maintain its reference topology when round-tripped.

• In your question you wrote // works ok with list<Model> but in fact this is not correct. When a List<T> is populated the new values are appended to the list, so Assert.AreSame(to_serialize[0], model); passes purely by luck. If you had additionally asserted Assert.AreSame(1, to_serialize.Count) it would have failed.

• While the converter will work for primitive keys such as string and int it may not work for key types that require JSON-specific conversion such as enum or DateTime.

• The converter is currently only implemented for Dictionary<TKey, TValue> and takes advantage of the fact that this type implements the non-generic IDictionary interface. It could be extended to other dictionary types such as SortedDictionary<TKey,TValue> if required.

Demo fiddle here.