Iterate IDictionary<string, string> with dynamic nested JSON as value in C#

Question

My application receives Kafka messages that contain a Dictionary<string,string> as one of the properties, and its values could be a nested (however dynamic) json string, and I need to iterate through this unknown json. I am struggling to find a logic and even the best data structure to do this iteration.

Examples of the dictionary (mocked data):

//could have complex nested json string as value
"reward":"{
  'xp':'200', 
  'gp':'150', 
  'loot':'{
    'item':'sword',
    'rarity': 'low'
  }'
}",
"achievement":"win_match"

// while other messages might be simple
"type":"generic_xp",
"percent":"100",
"status":"complete"

Serialized version of a real message:

"{\"player_stats\":\"{\\\"assist\\\":0,\\\"deaths\\\":0,\\\"kills\\\":0,\\\"team_index\\\":2}\",\"round_attr\":\"{\\\"max_player_count\\\":4,\\\"rdur\\\":0,\\\"round\\\":1,\\\"team_player_count\\\":{\\\"team_1\\\":1,\\\"team_2\\\":0},\\\"team_score\\\":0}\",\"custom\":\"{\\\"armor\\\":\\\"armor_pickup_lv2\\\",\\\"balance\\\":550,\\\"helmet\\\":\\\"helmet_pickup_lv2\\\",\\\"misc\\\":[{\\\"count\\\":48,\\\"item_id\\\":\\\"shotgun\\\"},{\\\"count\\\":120,\\\"item_id\\\":\\\"bullet\\\"},{\\\"count\\\":2,\\\"item_id\\\":\\\"health_pickup_combo_small\\\"},{\\\"count\\\":2,\\\"item_id\\\":\\\"health_pickup_health_small\\\"}],\\\"weapon_1\\\":\\\"mp_weapon_semipistol\\\",\\\"weapon_2\\\":\\\"mp_weapon_shotgun_pistol\\\"}\",\"gdur\":\"0\"}"

To complicate even more

• Create a model class is not an option because this json is completely dynamic
• Flatting the dictionary is not possible because the json may have duplicated key names, but under different hierarchy
• I cant request to change the Kafka message

What I am trying to do

The end user will define rules that I need to check if I find a match. For instance, a rule could be reward.xp == 200 or reward.loot.rarity == high or status == complete. These rules will be defined by the user so it cant be hardcoded, however I can decide with data structure to use to save them. So for each Kafka message, I have to iterate through that dictionary and try to find a match with the rules.

What I have tried

I ve tried JsonConvert.Deserialize to object, dynamic, ExpandoObject and none could handle the nested json hierarchy. They just got the 1st level correct. Same result with JObject.Parse as well.

Answer 1

Parse the JSON using whatever parser you like (I used Newtonsoft.Json).

Then recursively visit the hierarchy and copy each property to a flat list using the full path to each property value as a key. You can then iterate that flat list.

Edit: Comment requested supporting arrays, so this version does.

https://dotnetfiddle.net/6ykHT0

using System;
using Newtonsoft.Json.Linq;
using System.Linq;
using System.Collections.Generic;
                    
public class Program
{
    public static void Main()
    {
        string json = @"{
                        'reward': { 
                            'xp': '200', 
                            'gp': '150', 
                            'loot': {
                                'item': 'sword',
                                'rarity': 'low',
                                'blah': {
                                    'socks': 5
                                }
                            },
                            'arrayofint': [1,2,3,4],
                            'arrayofobj': [
                                {
                                    'foo': 'bar',
                                    'stuff': ['omg!', 'what?!']
                                },
                                {
                                    'foo': 'baz',
                                    'stuff': ['a', 'b']
                                }
                            ],
                            'arrayofarray': [
                                [1,2,3],
                                [4,5,6]
                            ],
                            'arrayofheterogenousjunk': [
                                'a',
                                2,
                                { 'objprop': 1 },
                                ['staahp!']
                            ]
                        },
                        'achievement': 'win_match'
                    }";
        
        JObject data = JObject.Parse(json);
        IList<string> nodes = flattenJSON(data);
        
        Console.WriteLine(string.Join(Environment.NewLine, nodes));
    }
    
    private static IList<string> flattenJSON(JToken token)
    {
        return _flattenJSON(token, new List<string>());
    }

    private static IList<string> _flattenJSON(JToken token, List<string> path)
    {
        var output = new List<string>();
        if (token.Type == JTokenType.Object)
        {
            // Output the object's child properties
            output.AddRange(token.Children().SelectMany(x => _flattenJSON(x, path)));
        }
        else if (token.Type == JTokenType.Array)
        {
            // Output each array element
            var arrayIndex = 0;
            foreach (var child in token.Children())
            {
                // Append the array index to the end of the last path segment - e.g. someProperty[n]
                var newPath = new List<string>(path);
                newPath[newPath.Count - 1] += "[" + arrayIndex++ + "]";
                output.AddRange(_flattenJSON(child, newPath));
            }
        }
        else if (token.Type == JTokenType.Property)
        {
            var prop = token as JProperty;
            // Insert the property name into the path
            output.AddRange(_flattenJSON(prop.Value, new List<string>(path) { prop.Name }));
        }
        else
        {
            // Join the path segments delimited with periods, followed by the literal value
            output.Add(string.Join(".", path) + " = " + token.ToString());
        }
        return output;
    }
}

Output:

reward.xp = 200
reward.gp = 150
reward.loot.item = sword
reward.loot.rarity = low
reward.loot.blah.socks = 5
reward.arrayofint[0] = 1
reward.arrayofint[1] = 2
reward.arrayofint[2] = 3
reward.arrayofint[3] = 4
reward.arrayofobj[0].foo = bar
reward.arrayofobj[0].stuff[0] = omg!
reward.arrayofobj[0].stuff[1] = what?!
reward.arrayofobj[1].foo = baz
reward.arrayofobj[1].stuff[0] = a
reward.arrayofobj[1].stuff[1] = b
reward.arrayofarray[0][0] = 1
reward.arrayofarray[0][1] = 2
reward.arrayofarray[0][2] = 3
reward.arrayofarray[1][0] = 4
reward.arrayofarray[1][1] = 5
reward.arrayofarray[1][2] = 6
reward.arrayofheterogenousjunk[0] = a
reward.arrayofheterogenousjunk[1] = 2
reward.arrayofheterogenousjunk[2].objprop = 1
reward.arrayofheterogenousjunk[3][0] = staahp!
achievement = win_match

PREVIOUS VERSION (NO ARRAY SUPPORT)

This doesn't properly support arrays - it will output the contents of a property that is an array as the raw JSON - i.e. it won't traverse into the array.

https://dotnetfiddle.net/yZbwul

public static void Main()
{
    string json = @"{
                    'reward': { 
                        'xp': '200', 
                        'gp': '150', 
                        'loot': {
                            'item': 'sword',
                            'rarity': 'low',
                            'blah': {
                                'socks': 5
                            }
                        }
                    },
                    'achievement': 'win_match'
                }";
    
    JObject data = JObject.Parse(json);
    IList<string> nodes = flattenJSON(data, new List<string>());
    
    Console.WriteLine(string.Join(Environment.NewLine, nodes));
}

private static IList<string> flattenJSON(JObject obj, IList<string> path)
{
    var output = new List<string>();
    foreach (var prop in obj.Properties())
    {
        if (prop.Value.Type == JTokenType.Object)
        {
            output.AddRange(flattenJSON(prop.Value as JObject, new List<string>(path){prop.Name}));
        }
        else
        {
            var s = string.Join(".", new List<string>(path) { prop.Name }) + " = " + prop.Value.ToString();
            output.Add(s);
        }
    }
    return output;
}

Output:

reward.xp = 200
reward.gp = 150
reward.loot.item = sword
reward.loot.rarity = low
reward.loot.blah.socks = 5
achievement = win_match