Query and update JSON within Python

Question

I need to be able to dynamically query a JSON object and then update or append it with values. Note the standard json package is not suitable for the task given the need to query and update an arbitrary set of values. I've found the following packages which support querying a JSON:

1. JsonPath_rw
2. objectpath
3. jmespath

However appear to only support querying the data (please correct if I'm misunderstanding!), not updating or appending. For example, given the following JSON:

{
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

If I wanted to append a new 'general' field nested under 'people' and update the id value to 'identifier' from 'id' how could I achieve this in Python using a query framework, such that it looks like:

{
  "people": [
    {
      "general": {
        "identifier": 100,
        "age": 20
      },
    {
      "general": {
        "identifier": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "identifier": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "identifier": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

Answer 1

I had the same problem.

I think you may find glom useful and more pythonic, besides you can still use jmespath to complement different uses.

Answer 2

Let's read in the structure:

dictPeople = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

It's a Python dictionary.

1.To add a new object under people, we use standard Python's dictionary functionality. No JSON querying is needed:

dictAdd = {
      "general": {
        "identifier": 100,
        "age": 30
      },
      "history": {
      }
    }
dictPeople['people'].append(dictAdd)

2.To update the field name which is in an unknown place in the structure, we can first use JSON querying (JSONPath) to localize it, using one of the libraries e.g. jsonpath-ng. After we get the smallest structure (a dictionary) with that field, we can update it using standard dictionary functionality of updating the key. After that, we apply the updated dictionary back to the JSON structure. The code is as follows:

import json 
from jsonpath_ng import jsonpath
from jsonpath_ng.ext import parse

# localize the fields to update based on JSONPath query
jsExp = parse("$..people[?general.id].general")
match = jsExp.find(dictPeople)
for obj in match:
    print(obj.value)
    # get the smallest structure with the field
    dictGeneral = obj.value # the structure is a dictionary
    # update the field
    dictGeneral['identifier'] = dictGeneral['id']
    del dictGeneral['id']
    # update JSON structure
    jsExp.update(dictPeople,dictGeneral)

print(dictPeople)

Answer 3

As Scott indicates, the object you've got there is not strictly speaking "a JSON object". It's a perfectly ordinary Python dict containing a perfectly ordinary list which contains perfectly ordinary dicts, so you can manipulate it using ordinary iteration/indexed assignment/etc, no frameworks required.

d = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

#add new person
d["people"].insert(0, {
    "general": {
        "id": 100,
        "age": 20,
    },
    "history": {}
})

#copy `id` over to `identifier` for each person,
#and delete `id`
for person in d["people"]:
    person["general"]["identifier"] = person["general"]["id"]
    del person["general"]["id"]

print(d)

Result:

{'people': [{'general': {'age': 20, 'identifier': 100}, 'history': {}}, {'general': {'age': 20, 'identifier': 100}, 'history': {}}, {'general': {'age': 30, 'identifier': 101}, 'history': {}}, {'general': {'age': 30, 'identifier': 100}, 'history': {}}]}

Adding whitespace, you get

{
    'people': [
        {
            'general': {
                'age': 20, 
                'identifier': 100
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 20, 
                'identifier': 100
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 30, 
                'identifier': 101
            }, 
            'history': {}
        }, 
        {
            'general': {
                'age': 30, 
                'identifier': 100
            }, 
            'history': {}
        }
    ]
}

---

Of course, this approach only works if you know the structure of the object. If the person sending you this data is allowed to change the structure whenever they want, then your code is very likely to break right away.

I think what you're hoping for is some kind of "smart parser" that can interpret an object with arbitrary structure and understand the conceptual meaning of each component. As far as I know, no such library exists, because it would need human-level intelligence (or better) to make effective guesses for anything other than simple cases.

...That said, you might be able to handle some level of changing structure as long as you can make certain guarantees about the data. Let's say that there will always be a "people" key that you want to append to, and there will always be an "id" key that you want to rename. If these facts stay constant, then you can walk through the dict and find the objects you need no matter where they are.

import copy

def find_key_item_pairs(obj, criteria):
    if isinstance(obj, dict):
        for key_and_value in obj.items():
            if criteria(key_and_value):
                yield key_and_value
            else:
                value = key_and_value[1]
                yield from find_key_item_pairs(value, criteria)
    elif isinstance(obj, list):
        for item in obj:
            yield from find_key_item_pairs(item, criteria)

d = {
  "people": [
    {
      "general": {
        "id": 100,
        "age": 20
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 101,
        "age": 30
      },
      "history": {
      }
    },
    {
      "general": {
        "id": 100,
        "age": 30
      },
      "history": {
      }
    }
  ]
}

#dynamically locate all people lists
for _, people_list in find_key_item_pairs(d, lambda kv: kv[0] == "people" and isinstance(kv[1], list)):
    #duplicate the first entry and insert
    people_list.insert(0, copy.deepcopy(people_list[0]))

#dynamically locate all dicts containing "id"
for _, person in find_key_item_pairs(d, lambda kv: isinstance(kv[1], dict) and "id" in kv[1]):
    #swap out "id" for "identifier"
    person["identifier"] = person["id"]
    del person["id"]

print(d)

Answer 4

JSON (as the name implies) is a way to represent a JavaScript Object. To do manipulation, the most appropriate thing would be to parse that representation into an actual object, manipulate that, then (if need be) create a new JSON representation of that updated object. (In fact, I would guess that these query packages do just that, possibly on just enough of the object to satisfy the query.)