Loop through all nested dictionary values?

Question

I'm trying to loop through a dictionary and print out all key value pairs where the value is not a nested dictionary. If the value is a dictionary I want to go into it and print out its key value pairs, etc.

I tried this. But it only works for the first two levels. I need it to work for any number of levels.

for k, v in d.iteritems():
    if type(v) is dict:
        for t, c in v.iteritems():
            print "{0} : {1}".format(t, c)

I also tried this. It still only prints one thing.

def printDict(d):
    for k, v in d.iteritems():
        if type(v) is dict:
            printDict(v)
        else:
            print "{0} : {1}".format(k, v)

Full test case:

Dictionary:

{u'xml': {u'config': {u'portstatus': {u'status': u'good'}, u'target': u'1'},
      u'port': u'11'}}

Result:

xml : {u'config': {u'portstatus': {u'status': u'good'}, u'target': u'1'}, u'port': u'11'}

Answer 1

Returns a tuple of each key and value and the key contains the full path

from typing import Mapping, Tuple, Iterator

def traverse_dict(nested: Mapping, parent_key="", keys_to_not_traverse_further=tuple()) -> Iterator[Tuple[str, str]]:
    """Each key is joined with it's parent using dot as a separator.

    Once a `parent_key` matches `keys_to_not_traverse_further` 
   it will no longer find its child dicts.
   """
    for key, value in nested.items():
        if isinstance(value, abc.Mapping) and key not in keys_to_not_traverse_further:
            yield from traverse_dict(value, f"{parent_key}.{key}", keys_to_not_traverse_further)
        else:
            yield f"{parent_key}.{key}", value

Let's test it

my_dict = {
    "isbn": "123-456-222",
    "author": {"lastname": "Doe", "firstname": "Jane"},
    "editor": {"lastname": "Smith", "firstname": "Jane"},
    "title": "The Ultimate Database Study Guide",
    "category": ["Non-Fiction", "Technology"],
    "first": {
        "second": {"third": {"fourth": {"blah": "yadda"}}},
        "fifth": {"sixth": "seventh"},
    },
}
for k, v in traverse_dict(my_dict):
    print(k, v)

Returns

.isbn 123-456-222
.author.lastname Doe
.author.firstname Jane
.editor.lastname Smith
.editor.firstname Jane
.title The Ultimate Database Study Guide
.category ['Non-Fiction', 'Technology']
.first.second.third.fourth.blah yadda
.first.fifth.sixth seventh

If you don't care about some child dicts e.g names in this case then use the keys_to_not_traverse_further

for k, v in traverse_dict(my_dict, parent_key="", keys_to_not_traverse_further=("author","editor")):
    print(k, v)

Returns

.isbn 123-456-222
.author {'lastname': 'Doe', 'firstname': 'Jane'}
.editor {'lastname': 'Smith', 'firstname': 'Jane'}
.title The Ultimate Database Study Guide
.category ['Non-Fiction', 'Technology']
.first.second.third.fourth.blah yadda
.first.fifth.sixth seventh

Answer 2

While the original solution from @Scharron is beautiful and simple, it cannot handle the list very well:

def myprint(d):
    for k, v in d.items():
        if isinstance(v, dict):
            myprint(v)
        else:
            print("{0} : {1}".format(k, v))

So this code can be slightly modified like this to handle list in elements:

def myprint(d):
    for k, v in d.items():
        if isinstance(v, dict):
            myprint(v)
        elif isinstance(v, list):
            for i in v:
                myprint(i)
        else:
            print("{0} : {1}".format(k, v))

Answer 3

There are potential problems if you write your own recursive implementation or the iterative equivalent with stack. See this example:

dic = {}
dic["key1"] = {}
dic["key1"]["key1.1"] = "value1"
dic["key2"]  = {}
dic["key2"]["key2.1"] = "value2"
dic["key2"]["key2.2"] = dic["key1"]
dic["key2"]["key2.3"] = dic

In the normal sense, nested dictionary will be a n-nary tree like data structure. But the definition doesn't exclude the possibility of a cross edge or even a back edge (thus no longer a tree). For instance, here key2.2 holds to the dictionary from key1, key2.3 points to the entire dictionary(back edge/cycle). When there is a back edge(cycle), the stack/recursion will run infinitely.

            root<-------back edge
          /      \           |
       _key1   __key2__      |
      /       /   \    \     |
 |->key1.1 key2.1 key2.2 key2.3
 |   /       |      |
 | value1  value2   |
 |                  | 
cross edge----------|

If you print this dictionary with this implementation from Scharron

def myprint(d):
    for k, v in d.items():
        if isinstance(v, dict):
            myprint(v)
        else:
            print "{0} : {1}".format(k, v)
            

You would see this error:

> RuntimeError: maximum recursion depth exceeded while calling a Python object

The same goes with the implementation from senderle.

Similarly, you get an infinite loop with this implementation from Fred Foo:

def myprint(d):
    stack = list(d.items())
    while stack:
        k, v = stack.pop()
        if isinstance(v, dict):
            stack.extend(v.items())
        else:
            print("%s: %s" % (k, v))

However, Python actually detects cycles in nested dictionary:

print dic
{'key2': {'key2.1': 'value2', 'key2.3': {...}, 
       'key2.2': {'key1.1': 'value1'}}, 'key1': {'key1.1': 'value1'}}

"{...}" is where a cycle is detected.

As requested by Moondra this is a way to avoid cycles (DFS):

def myprint(d): 
    stack = list(d.items()) 
    visited = set() 
    while stack: 
        k, v = stack.pop() 
        if isinstance(v, dict): 
            if k not in visited: 
                stack.extend(v.items()) 
        else: 
            print("%s: %s" % (k, v)) 
        visited.add(k)

Answer 4

For a ready-made solution install ndicts

pip install ndicts

Import a NestedDict in your script

from ndicts.ndicts import NestedDict

Initialize

dictionary = {
    u'xml': {
        u'config': {
            u'portstatus': {u'status': u'good'}, 
            u'target': u'1'
        },
    u'port': u'11'
    }
}

nd = NestedDict(dictionary)

Iterate

for key, value in nd.items():
    print(key, value)

Answer 5

You can print recursively with a dictionary comprehension:

def print_key_pairs(d):
    {k: print_key_pairs(v) if isinstance(v, dict) else print(f'{k}: {v}') for k, v in d.items()}

For your test case this is the output:

>>> print_key_pairs({u'xml': {u'config': {u'portstatus': {u'status': u'good'}, u'target': u'1'}, u'port': u'11'}})
status: good
target: 1
port: 11

Answer 6

Nested dictionaries looping using isinstance() and yield function. **isinstance is afunction that returns the given input and reference is true or false as in below case dict is true so it go for iteration. **Yield is used to return from a function without destroying the states of its local variable and when the function is called, the execution starts from the last yield statement. Any function that contains a yield keyword is termed a generator.

students= {'emp1': {'name': 'Bob', 'job': 'Mgr'},
     'emp2': {'name': 'Kim', 'job': 'Dev','emp3': {'namee': 'Saam', 'j0ob': 'Deev'}},
     'emp4': {'name': 'Sam', 'job': 'Dev'}}
def nested_dict_pairs_iterator(dict_obj):
     for key, value in dict_obj.items():
        # Check if value is of dict type
        if isinstance(value, dict):
            # If value is dict then iterate over all its values
            for pair in  nested_dict_pairs_iterator(value):
                yield (key, *pair)
        else:
            # If value is not dict type then yield the value
            yield (key, value)
for pair in nested_dict_pairs_iterator(students):
    print(pair)

Answer 7

Your question already has been answered well, but I recommend using isinstance(d, collections.Mapping) instead of isinstance(d, dict). It works for dict(), collections.OrderedDict(), and collections.UserDict().

The generally correct version is:

def myprint(d):
    for k, v in d.items():
        if isinstance(v, collections.Mapping):
            myprint(v)
        else:
            print("{0} : {1}".format(k, v))

Answer 8

As said by Niklas, you need recursion, i.e. you want to define a function to print your dict, and if the value is a dict, you want to call your print function using this new dict.

Something like :

def myprint(d):
    for k, v in d.items():
        if isinstance(v, dict):
            myprint(v)
        else:
            print("{0} : {1}".format(k, v))

Answer 9

These answers work for only 2 levels of sub-dictionaries. For more try this:

nested_dict = {'dictA': {'key_1': 'value_1', 'key_1A': 'value_1A','key_1Asub1': {'Asub1': 'Asub1_val', 'sub_subA1': {'sub_subA1_key':'sub_subA1_val'}}},
                'dictB': {'key_2': 'value_2'},
                1: {'key_3': 'value_3', 'key_3A': 'value_3A'}}

def print_dict(dictionary):
    dictionary_array = [dictionary]
    for sub_dictionary in dictionary_array:
        if type(sub_dictionary) is dict:
            for key, value in sub_dictionary.items():
                print("key=", key)
                print("value", value)
                if type(value) is dict:
                    dictionary_array.append(value)



print_dict(nested_dict)

Answer 10

I am using the following code to print all the values of a nested dictionary, taking into account where the value could be a list containing dictionaries. This was useful to me when parsing a JSON file into a dictionary and needing to quickly check whether any of its values are None.

    d = {
            "user": 10,
            "time": "2017-03-15T14:02:49.301000",
            "metadata": [
                {"foo": "bar"},
                "some_string"
            ]
        }


    def print_nested(d):
        if isinstance(d, dict):
            for k, v in d.items():
                print_nested(v)
        elif hasattr(d, '__iter__') and not isinstance(d, str):
            for item in d:
                print_nested(item)
        elif isinstance(d, str):
            print(d)

        else:
            print(d)

    print_nested(d)

Output:

    10
    2017-03-15T14:02:49.301000
    bar
    some_string

Answer 11

I find this approach a bit more flexible, here you just providing generator function that emits key, value pairs and can be easily extended to also iterate over lists.

def traverse(value, key=None):
    if isinstance(value, dict):
        for k, v in value.items():
            yield from traverse(v, k)
    else:
        yield key, value

Then you can write your own myprint function, then would print those key value pairs.

def myprint(d):
    for k, v in traverse(d):
        print(f"{k} : {v}")

A test:

myprint({
    'xml': {
        'config': {
            'portstatus': {
                'status': 'good',
            },
            'target': '1',
        },
        'port': '11',
    },
})

Output:

status : good
target : 1
port : 11

I tested this on Python 3.6.

Answer 12

Since a dict is iterable, you can apply the classic nested container iterable formula to this problem with only a couple of minor changes. Here's a Python 2 version (see below for 3):

import collections
def nested_dict_iter(nested):
    for key, value in nested.iteritems():
        if isinstance(value, collections.Mapping):
            for inner_key, inner_value in nested_dict_iter(value):
                yield inner_key, inner_value
        else:
            yield key, value

Test:

list(nested_dict_iter({'a':{'b':{'c':1, 'd':2}, 
                            'e':{'f':3, 'g':4}}, 
                       'h':{'i':5, 'j':6}}))
# output: [('g', 4), ('f', 3), ('c', 1), ('d', 2), ('i', 5), ('j', 6)]

In Python 2, It might be possible to create a custom Mapping that qualifies as a Mapping but doesn't contain iteritems, in which case this will fail. The docs don't indicate that iteritems is required for a Mapping; on the other hand, the source gives Mapping types an iteritems method. So for custom Mappings, inherit from collections.Mapping explicitly just in case.

In Python 3, there are a number of improvements to be made. As of Python 3.3, abstract base classes live in collections.abc. They remain in collections too for backwards compatibility, but it's nicer having our abstract base classes together in one namespace. So this imports abc from collections. Python 3.3 also adds yield from, which is designed for just these sorts of situations. This is not empty syntactic sugar; it may lead to faster code and more sensible interactions with coroutines.

from collections import abc
def nested_dict_iter(nested):
    for key, value in nested.items():
        if isinstance(value, abc.Mapping):
            yield from nested_dict_iter(value)
        else:
            yield key, value

Answer 13

Slightly different version I wrote that keeps track of the keys along the way to get there

def print_dict(v, prefix=''):
    if isinstance(v, dict):
        for k, v2 in v.items():
            p2 = "{}['{}']".format(prefix, k)
            print_dict(v2, p2)
    elif isinstance(v, list):
        for i, v2 in enumerate(v):
            p2 = "{}[{}]".format(prefix, i)
            print_dict(v2, p2)
    else:
        print('{} = {}'.format(prefix, repr(v)))

On your data, it'll print

data['xml']['config']['portstatus']['status'] = u'good'
data['xml']['config']['target'] = u'1'
data['xml']['port'] = u'11'

It's also easy to modify it to track the prefix as a tuple of keys rather than a string if you need it that way.

Answer 14

Here is pythonic way to do it. This function will allow you to loop through key-value pair in all the levels. It does not save the whole thing to the memory but rather walks through the dict as you loop through it

def recursive_items(dictionary):
    for key, value in dictionary.items():
        if type(value) is dict:
            yield (key, value)
            yield from recursive_items(value)
        else:
            yield (key, value)

a = {'a': {1: {1: 2, 3: 4}, 2: {5: 6}}}

for key, value in recursive_items(a):
    print(key, value)

Prints

a {1: {1: 2, 3: 4}, 2: {5: 6}}
1 {1: 2, 3: 4}
1 2
3 4
2 {5: 6}
5 6

Answer 15

Here's a modified version of Fred Foo's answer for Python 2. In the original response, only the deepest level of nesting is output. If you output the keys as lists, you can keep the keys for all levels, although to reference them you need to reference a list of lists.

Here's the function:

def NestIter(nested):
    for key, value in nested.iteritems():
        if isinstance(value, collections.Mapping):
            for inner_key, inner_value in NestIter(value):
                yield [key, inner_key], inner_value
        else:
            yield [key],value

To reference the keys:

for keys, vals in mynested: 
    print(mynested[keys[0]][keys[1][0]][keys[1][1][0]])

for a three-level dictionary.

You need to know the number of levels before to access multiple keys and the number of levels should be constant (it may be possible to add a small bit of script to check the number of nesting levels when iterating through values, but I haven't yet looked at this).

Answer 16

A alternative solution to work with lists based on Scharron's solution

def myprint(d):
    my_list = d.iteritems() if isinstance(d, dict) else enumerate(d)

    for k, v in my_list:
        if isinstance(v, dict) or isinstance(v, list):
            myprint(v)
        else:
            print u"{0} : {1}".format(k, v)

Answer 17

Alternative iterative solution:

def myprint(d):
    stack = d.items()
    while stack:
        k, v = stack.pop()
        if isinstance(v, dict):
            stack.extend(v.iteritems())
        else:
            print("%s: %s" % (k, v))

Answer 18

Iterative solution as an alternative:

def traverse_nested_dict(d):
    iters = [d.iteritems()]

    while iters:
        it = iters.pop()
        try:
            k, v = it.next()
        except StopIteration:
            continue

        iters.append(it)

        if isinstance(v, dict):
            iters.append(v.iteritems())
        else:
            yield k, v


d = {"a": 1, "b": 2, "c": {"d": 3, "e": {"f": 4}}}
for k, v in traverse_nested_dict(d):
    print k, v