Deserializing Indexed Properties into an Array

Question

Say I have some Json that will come in a packet like this:

{
    "LinkType1": "google",
    "LinkUrl1": "https://plus.google.com/test",
    "LinkShow1": 1,

    "LinkType2": "facebook",
    "LinkUrl2": "https://www.facebook.com/test",
    "LinkShow2": 0,

    "LinkType3": "linkedin",
    "LinkUrl3": "http://www.linkedin.com/test",
    "LinkShow3": 1,

    "count": 3,
    "errorCode": 0,
    "errorMessage": "Success"
}

Notice how everything comes back as the same property, but with an index on it?

I would love to be able to deserialize that data as though it was an array instead of single properties. What would be the best method for deserializing this into the classes below? I'm using the Newtonsoft Json library for serialization, so a solution using that would be preferred.

    public class LinksResult
    {
        public List<LinkData> Links { get; set; } 

        [JsonProperty("count")]
        public int Count { get; set; }

        [JsonProperty("errorCode")]
        public int ErrorCode { get; set; }

        [JsonProperty("errorMessage")]
        public string ErrorMessage { get; set; }
    }

    public class LinkData
    {
        public string LinkType { get; set; }
        public string LinkUrl { get; set; }
        public bool LinkShow { get; set; }
    }

Answer 1

Below is a solution I came up with that works pretty well.

Had some sample JSON similar to below. JSON was very flat and you can see the indexed Check/Transfer properties. I wanted to allow NewtonSoft to do as much of the heavy lifting as possible.

{
  "id": "209348",
  "Check__00__amount": 10000,
  "Check__00__payableTo": "ABC Company",
  "Check__00__receivedFrom": "Mike",
  "Check__01__amount": 20000,
  "Check__01__payableTo": "XYZ Company",
  "Check__01__receivedFrom": "Jim",
  "Transfer00__Amount": 50000.0,
  "Transfer00__CompanyTransferringFrom": "DEF Company",
  "Transfer00__Type": "Partial",
  "Transfer01__Amount": 55000.0,
  "Transfer01__CompanyTransferringFrom": "GHI Company",
  "Transfer01__Type": "Full"
}

Types to deserialize to. Top level object Transaction with two list properties for the checks and transfers. I'm using a custom converter and a custom attribute.

[JsonConverter(typeof(TestConverter))]
public class Transaction
{
    [JsonProperty("id")]
    public int Id { get; set; } = default!;
    
    [RegexIndexedPropertiesToList(@"Transfer\d+__")]
    public List<Transfer> Transfers { get; set; } = default!;
    
    [RegexIndexedPropertiesToList(@"Check__\d+__")]
    public List<Check> Checks { get; set; } = default!;
}

public class Check
{
    
    [JsonProperty("amount")]
    public decimal Amount { get; set; }
    [JsonProperty("payableTo")]
    public string PayableTo { get; set; } = default!;
    [JsonProperty("receivedFrom")]
    public string ReceivedFrom { get; set; } = default!;
    
}

public class Transfer
{
    [JsonProperty("Amount")]
    public decimal Amount { get; set; }
    [JsonProperty("CompanyTransferringFrom")]
    public string CompanyTransferringFrom { get; set; } = default!;
    [JsonProperty("Type")]
    public string Type { get; set; } = default!;
}

custom attribute. Allows for the setting of the regex. This regex should be able to Match the properties that you want to turn into a list. You can see it decorated on the Checks/List properties in the Transaction class.

[AttributeUsage(AttributeTargets.Property, AllowMultiple = false)]
public class RegexIndexedPropertiesToListAttribute : Attribute
{
    public string Regex { get; }
    public RegexIndexedPropertiesToListAttribute(string regex)
    {
        Regex = regex;
    }
}

The converter

public class TestConverter : JsonConverter
{

    public override void WriteJson(JsonWriter writer, object? value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override object? ReadJson(JsonReader reader, Type objectType, object? existingValue, JsonSerializer serializer)
    {
        var obj = JObject.Load(reader);
            
        //get a new instance and populate it to prevent infinite recursion on the converter, this will allow default serializer behavior to work
        var returnable = Activator.CreateInstance(objectType) ?? throw new InvalidOperationException("could not create instance");
        JsonConvert.PopulateObject(obj.ToString(), returnable); //use regular deserialization

        IndexedPropertiesToList(returnable, obj);

        return returnable;
    }

    

    private static void IndexedPropertiesToList(object returnable, JObject obj)
    {
        //get all the index to list properties
        var propsForConversion = returnable.GetType().GetProperties()
            .Where(x => x.IsDefined(typeof(RegexIndexedPropertiesToListAttribute), false));

        foreach (var prop in propsForConversion)
        {
            var attribute = (prop.GetCustomAttributes(typeof(RegexIndexedPropertiesToListAttribute), false).FirstOrDefault() as RegexIndexedPropertiesToListAttribute)
                            ?? throw new Exception("attribute not found");

            //assume the prperty to be set is a list
            var list = Activator.CreateInstance(prop.PropertyType) as IList ??
                       throw new InvalidOperationException("could not create instance");
            
            var regex = new Regex(attribute.Regex);

            //get the properties that match the regex and then group them using the regex as a prefix
            var matchedProperties = obj.Properties().Where(x => regex.IsMatch(x.Path)).ToList();
            var groups = matchedProperties.GroupBy(x => regex.Match(x.Path).Value).ToList();

            foreach (var group in groups)
            {
                var newObj = new JObject(); //create a new jobject will use this to deserialize the properties into type so that normal deserialization works
                foreach (var property in group)
                {
                    var name = property.Name.Replace(group.Key, "");
                    newObj.Add(name, property.Value); //add the property to the new object with no index
                }

                //assumes the List is of a generic type
                var genericType = prop.PropertyType.GenericTypeArguments[0];
                
                var instance = newObj.ToObject(genericType) ??
                               throw new InvalidOperationException("could not deserialize");
                list.Add(instance);
            }

            //set the constructed list of deserialized objects to the property
            prop.SetValue(returnable, list);
        }
    }

    public override bool CanConvert(Type objectType)
    {
        throw new NotImplementedException();
    }
}

First the converter populates the top level in this case the Transaction. Then it looks for properties that are decorated with the attribute. It uses that regex to match fields from the JSON and then groups by the match portion, Check__00__ and Check__01__ would be two groupings with 3 JTokens each for the corresponding properties. From there it builds a new JObject with the grouped fields. The JObject will have the regex matched portion of the name removed which allows for the usage of JsonProperty attributes for mapping. Then it uses the default Newtonsoft deserialization to get a new instance of the List type and adds it to the list, then sets the List property on the top level instance.

Its worked well for what I've needed it for.

The good

Uses regular NewtonSoft deserialzation only requires the custom converter and the attribute Seems to execute pretty quickly on some sizable payloads.

What I have above assumes it is deeling with a List and further adaptation would be needed if that weren't the case. It wouldn't be able to handle deeper nesting, though I think it could be added if needed, and probably wouldn't play well with List of primitives, though I think that could be handled as well with a bit of additional code.

Answer 2

Brian's answer was very good and it got me 80% of the way to where I wanted to be. However it's not a very good implementation to use over and over again if this sort of pattern happens on many different objects.

I made something more generic. An interface that a "Page" would have.

public interface IPage<TItem>
{
    int Count { get; set; }
    List<TItem> PageItems { get; set; }
}

Then the Page converter itself.

public class PageConverter<TPage, TItem> : JsonConverter
        where TPage : IPage<TItem>, new()
        where TItem : new()
{
    private readonly Regex _numberPostfixRegex = new Regex(@"\d+$");

    public override bool CanWrite
    {
        get { return false; }
    }

    public override bool CanConvert(Type objectType)
    {
        return (objectType == typeof(TPage));
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        var obj = serializer.Deserialize<JObject>(reader);
        var page = new TPage();
        serializer.Populate(obj.CreateReader(), page); //Loads everything that isn't a part of the items. 
        page.PageItems = new List<TItem>();

        for (int i = 1; i <= page.Count; i++)
        {
            string index = i.ToString();

            //Find all properties that have a number at the end, then any of those that are the same number as the current index.
            //Put those in a new JObject.
            var jsonItem = new JObject();
            foreach (var prop in obj.Properties().Where(p => _numberPostfixRegex.Match(p.Name).Value == index))
            {
                jsonItem[_numberPostfixRegex.Replace(prop.Name, "")] = prop.Value;
            }

            //Deserialize and add to the list.
            TItem item = jsonItem.ToObject<TItem>(serializer);
            page.PageItems.Add(item);
        }

        return page;
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

So then all that's needed is to implement it on the links result:

[JsonConverter(typeof(PageConverter<LinksResult, LinkData>))]
public class LinksResult : IPage<LinkData>
{
    public int Count { get; set; }

    public List<LinkData> PageItems { get; set; }
}

I figured out you can control the serialization of capitalization with JsonSerializerSettings, so best leave that detail up to the chosen serializer, not my converter.

Fiddle here: https://dotnetfiddle.net/7KhwYY

Answer 3

You can use a custom JsonConverter to deserialize the JSON data into the structure that you want. Here is what the code for the converter might look like.

class LinksResultConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return (objectType == typeof(LinksResult));
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        JObject obj = JObject.Load(reader);
        LinksResult result = new LinksResult();
        result.Count = (int)obj["count"];
        result.ErrorCode = (int)obj["errorCode"];
        result.ErrorMessage = (string)obj["errorMessage"];
        result.Links = new List<LinkData>();

        for (int i = 1; i <= result.Count; i++)
        {
            string index = i.ToString();
            LinkData link = new LinkData();
            link.LinkType = (string)obj["LinkType" + index];
            link.LinkUrl = (string)obj["LinkUrl" + index];
            link.LinkShow = (int)obj["LinkShow" + index] == 1;
            result.Links.Add(link);
        }

        return result;
    }

    public override bool CanWrite
    {
        get { return false; }
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

To use the converter, just add a [JsonConverter] attribute to your LinksResult class as shown below. (Note that you don't need the [JsonProperty] attributes with this approach, since the mapping between JSON property names and the actual class members is handled directly by the converter.)

[JsonConverter(typeof(LinksResultConverter))]
public class LinksResult
{
    public List<LinkData> Links { get; set; }
    public int Count { get; set; }
    public int ErrorCode { get; set; }
    public string ErrorMessage { get; set; }
}

Then, you can deserialize like this:

LinksResult result = JsonConvert.DeserializeObject<LinksResult>(json);

Fiddle: https://dotnetfiddle.net/56b34H

Answer 4

Here is similar solution you may apply. See Serialize json to an object with catch all dictionary property The answer by David Hoerster.