CODEWITHSUNDEEP
pythonpandasdataframedictionary
webe3
webe3

Reputation: 103

how to convert a python dict to a pandas dataframe

I was using DataFrame.from_dict to convert a python dictionary to a pandas dataframe. It does exactly what I want if the items in the dictionary are in a certain order. I have a parameters dictionary where some items are single values, some are lists, some are dictionaries. As long as I don't have a list or dictionary in the first position of the dict it works perfectly. If I have a list or dict in the first position, it does not work.

>>> import pandas as pd
>>> my_dict = {'a': 1, 'b': [1, 2, 3], 'c': {'x': 'aa', 'y': 'bb'}}
>>> my_dict
{'a': 1, 'b': [1, 2, 3], 'c': {'x': 'aa', 'y': 'bb'}}
>>> pd.DataFrame.from_dict(my_dict, orient='index').reset_index().rename(columns={'index': 'prop_name', 0: 'prop_value'})
  prop_name              prop_value
0         a                       1
1         b               [1, 2, 3]
2         c  {'x': 'aa', 'y': 'bb'}
>>> my_dict2 = {'c': {'x': 'aa', 'y': 'bb'}, 'a': 1, 'b': [1, 2, 3]}
>>> pd.DataFrame.from_dict(my_dict2, orient='index').reset_index().rename(columns={'index': 'prop_name', 0: 'prop_value'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 1300, in from_dict
    data = _from_nested_dict(data)
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 9281, in _from_nested_dict
    for col, v in s.items():
AttributeError: 'int' object has no attribute 'items'
>>> my_dict3 = {'b': [1, 2, 3], 'c': {'x': 'aa', 'y': 'bb'}, 'a': 1 }
>>> pd.DataFrame.from_dict(my_dict3, orient='index').reset_index().rename(columns={'index': 'prop_name', 0: 'prop_value'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 1309, in from_dict
    return cls(data, index=index, columns=columns, dtype=dtype)
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 509, in __init__
    arrays, columns = to_arrays(data, columns, dtype=dtype)
  File "/usr/lib/python3/dist-packages/pandas/core/internals/construction.py", line 524, in to_arrays
    return _list_to_arrays(data, columns, coerce_float=coerce_float, dtype=dtype)
  File "/usr/lib/python3/dist-packages/pandas/core/internals/construction.py", line 561, in _list_to_arrays
    content = list(lib.to_object_array(data).T)
  File "pandas/_libs/lib.pyx", line 2448, in pandas._libs.lib.to_object_array
TypeError: object of type 'int' has no len()
>>>

It is easy to see in the pandas code why the errors occur. What is the better way to do this so that I don't get the errors depending on what is in the first position? It does exactly what I want otherwise.

Upvotes: 3

Views: 7530

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195438

You can create dataframe directly, using dictionary:

my_dict2 = {"c": {"x": "aa", "y": "bb"}, "a": 1, "b": [1, 2, 3]}
df = pd.DataFrame(
    {"prop_name": my_dict2.keys(), "prop_value": my_dict2.values()}
)
print(df)

Prints:

  prop_name              prop_value
0         c  {'x': 'aa', 'y': 'bb'}
1         a                       1
2         b               [1, 2, 3]

With my_dict2 = {"a": 1, "b": [1, 2, 3], "c": {"x": "aa", "y": "bb"}} this produces:

  prop_name              prop_value
0         a                       1
1         b               [1, 2, 3]
2         c  {'x': 'aa', 'y': 'bb'}

Note: as @TrentonMcKinney said in the comments, the behaviour of how the dataframe is constructed depends on the first item of the dictionary (source):

                if isinstance(list(data.values())[0], (Series, dict)):
                    data = _from_nested_dict(data)
                else:
                    data, index = list(data.values()), list(data.keys())

so pd.DataFrame.from_dict({"b": 1, "a": [1, 2, 3]}, orient="index") succeeds and pd.DataFrame.from_dict({"a": [1, 2, 3], "b": 1},orient="index") produces an error.

Upvotes: 4

Related Questions