access a dictionary value with dynamic path using in python

Question

I have this piece of code to extract a value from a dictionary object

extracted_value = response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]
extracted_value = response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["bill"]["dueDate"]

These are just two samples but I have a dozen of these with different key/path combinations. How can I just call these using a module and do something like this

def get_value_from_content (response_content, my_path):
    # how can I use the value in my_path as the key instead of this hard coded path ?
    extracted_value =  response_content["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]
    #extracted_value = response_content using my_path is what I would like to do
    return extracted_value

#I get this from a REST API call but skipping the code here and just hard coding to ask the question here
response_content = {u'retrievePolicyBillingSummariesResponse': {u'billingSummaries': {u'policyBillingSummary': [{u'policy': {u'status': u'A', u'policyNumber': u'xyz123', u'writingCompany': u'FBI', u'renewalFlag': u'false', u'convertedRenewalOffer': u'false', u'termExpirationDate': u'2017-06-26', u'lineOfBusiness': u'PC', u'termEffectiveDate': u'2016-06-26', u'riskState': u'CA', u'insureds': {u'namedInsuredSummary': [{u'preferredPostalAddress': {u'streetAddressLine': u'1 disney', u'cityName': u'palo alto', u'zipCode': u'94100', u'isoRegionCode': u'CA'}, u'name': {u'lastName': u'DOE', u'fullName': u'john doe', u'firstName': u'john'}}]}, u'additionalInterests': {u'additionalInterest': [{u'billTo': u'N', u'name': {u'partyType': u'Organization'}}]}, u'type': u'PA', u'statusDescription': u'Active', u'dataSource': u'from_heaven'}, u'billingSummary': {u'paymentRestriction': u'false', u'nextInstallmentAmount': u'0.00', u'bill': {u'installmentNumber': u'1', u'statementDate': u'2016-06-26', u'paymentPlan': u'Direct', u'installmentAmount': u'12.00', u'totalBillAmountDue': u'1.76', u'previousBalance': u'0.00', u'dueDate': u'2016-06-26', u'billingPlan': u'ANN'}, u'lastPayment': {u'status': u'A'}, u'currentBalance': u'16.66', u'payOffAmount': u'15.66', u'isRestrictedToPay': u'false'}}]}}}

my_path = '["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["lastPayment"]["status"]'
get_extracted_item = get_value_from_content(response_content,my_path)

my_path = '["retrievePolicyBillingSummariesResponse"]["billingSummaries"]["policyBillingSummary"][0]["billingSummary"]["bill"]["dueDate"]'
get_extracted_item = get_value_from_content(response_content,my_path)

Answer 1

What about using an iterative solution, as follows:

response_content = {u'retrievePolicyBillingSummariesResponse': {u'billingSummaries': {u'policyBillingSummary': [{u'policy': {u'status': u'A', u'policyNumber': u'xyz123', u'writingCompany': u'FBI', u'renewalFlag': u'false', u'convertedRenewalOffer': u'false', u'termExpirationDate': u'2017-06-26', u'lineOfBusiness': u'PC', u'termEffectiveDate': u'2016-06-26', u'riskState': u'CA', u'insureds': {u'namedInsuredSummary': [{u'preferredPostalAddress': {u'streetAddressLine': u'1 disney', u'cityName': u'palo alto', u'zipCode': u'94100', u'isoRegionCode': u'CA'}, u'name': {u'lastName': u'DOE', u'fullName': u'john doe', u'firstName': u'john'}}]}, u'additionalInterests': {u'additionalInterest': [{u'billTo': u'N', u'name': {u'partyType': u'Organization'}}]}, u'type': u'PA', u'statusDescription': u'Active', u'dataSource': u'from_heaven'}, u'billingSummary': {u'paymentRestriction': u'false', u'nextInstallmentAmount': u'0.00', u'bill': {u'installmentNumber': u'1', u'statementDate': u'2016-06-26', u'paymentPlan': u'Direct', u'installmentAmount': u'12.00', u'totalBillAmountDue': u'1.76', u'previousBalance': u'0.00', u'dueDate': u'2016-06-26', u'billingPlan': u'ANN'}, u'lastPayment': {u'status': u'A'}, u'currentBalance': u'16.66', u'payOffAmount': u'15.66', u'isRestrictedToPay': u'false'}}]}}}

my_path = [
    "retrievePolicyBillingSummariesResponse",
    "billingSummaries",
    "policyBillingSummary",
    0,
    "billingSummary",
    "lastPayment",
    "status"
]

def get_value_from_content(extraction, my_path):
    for el in my_path:
        if isinstance(extraction, dict):
            extraction = extraction.get(el, extraction)
        else:
            extraction = extraction[el]
    return extraction

extraction = get_value_from_content(response_content, my_path)
print(extraction)

The function get_value_from_content can even be shorter than before, i.e.

def get_value_from_content(extraction, my_path):
    for el in my_path:
        extraction = extraction[el]
    return extraction

This last version of get_value_from_content is nonetheless more prone to throw exceptions if one has, e.g. misread the chaining of path components. It thus remains to be determined whether the string object my_path is human- or machine-made.

Which, in the two cases, returns "A". Tested in Python 2 and 3. Note also that I favor an iterative solution over a recursive one since the former is usually faster than the latter. Between 20% and 40% faster in the present case.

That being said, this does not address the question, since my_path is originally not a list object, but a string object. To adress the question, one would first convert this string into a list of keys/indexes, and then process it as mentioned above. Or, as @Minji does, one may want to use the python built-in function eval. Even if using this function is said to be a bad practice, I wonder to what extent the use of eval cannot be described in this situation as the best way to go.

Answer 2

first of all it will be easier to write small utility function like

def extract_from_dictionary(dictionary, *keys_or_indexes):
    value = dictionary
    for key_or_index in keys_or_indexes:
        value = value[key_or_index]
    return value

as we can see from your example there is an object called billingSummary that appears in required paths, so we can avoid boilerplate with

def get_billing_summary(response_content):
    return extract_from_dictionary(
        response_content,
        "retrievePolicyBillingSummariesResponse",
        "billingSummaries",
        "policyBillingSummary",
        0,
        "billingSummary")

then we can simply write

def get_value_from_content(response_content, *keys):
    billing_summary = get_billing_summary(response_content)
    extracted_value = extract_from_dictionary(billing_summary,
                                              *keys)
    return extracted_value

and obtain required objects like

last_payment_status = get_value_from_content(response_content,
                                             "lastPayment",
                                             "status")
bill_due_date = get_value_from_content(response_content,
                                       "bill",
                                       "dueDate")
print("last_payment_status:", last_payment_status)
print("bill_due_date:", bill_due_date)

gives us

last_payment_status: A
bill_due_date: 2016-06-26

Answer 3

eval() interprets a string as code

def get_value_from_content (response_content, my_path):
    # string is arguments name
    item = "response_content" + my_path
    return eval(item)

Answer 4

What about a recursive function?

def get_value(response, index):
    if len(index) > 1:
        return get_value(response[index[0]], index[1:])
    else:
        return response[index[0]]

index = ["retrievePolicyBillingSummariesResponse", "billingSummaries", "policyBillingSummary", 0, "billingSummary", "lastPayment", "status"]

get_value(response_content, index)