CODEWITHSUNDEEP
pythoncsvpandasnumpydictionary
user3543300
user3543300

Reputation: 507

Save and Read a Python Dictionary to/from CSV

I want to perform a simple operation, store a python dictionary into a csv file and then later read it from the file as a dictionary.

My dictionary maps a string to a list of numpy arrays, for example:

d = {'x': [array([2, 3, 4]), array([5, 6, 7])], 'y': [array([1, 2, 3]), array([4, 5, 6])]

So I would like to store this dictionary as a csv and then recreate the dictionary from the file in a different program.

I've tried using python's csv module to write the dictionary to a csv file but had trouble storing the list of multidimensional numpy arrays properly. When I used the module to read it back it read back blank rows in the csv file.

I also tried to use pandas, but wasn't able to figure out how to use the read_csv() method to read a list of numpy arrays back from the dictionary.

Upvotes: 0

Views: 917

Answers (1)

hpaulj
hpaulj

Reputation: 231665

In [610]: d = {'x': [np.array([2, 3, 4]), np.array([5, 6, 7])], 'y': [np.array(
     ...: [1, 2, 3]), np.array([4, 5, 6])]}
In [611]: d
Out[611]: 
{'x': [array([2, 3, 4]), array([5, 6, 7])],
 'y': [array([1, 2, 3]), array([4, 5, 6])]}

In [613]: np.save('test.npy',d)
In [614]: np.load('test.npy')
Out[614]: array({'x': [array([2, 3, 4]), array([5, 6, 7])], 'y': [array([1, 2, 3]), array([4, 5, 6])]}, dtype=object)

So calling save on the dictionary, it wraps the dictionary in a object type array, and then saves that. And elements of an object array are saved with their respective pickle method. So it pickles a dictionary, and the lists within the dictionary. Finally the arrays within the lists pickled with a version of the np.save. Lots of nesting. But it works.

And item can be used to pull the dictionary out of the object array:

In [616]: dd=np.load('test.npy').item()
In [617]: dd
Out[617]: 
{'x': [array([2, 3, 4]), array([5, 6, 7])],
 'y': [array([1, 2, 3]), array([4, 5, 6])]}

And using pickle directly:

In [626]: pickle.dump(d, open('test.pkl','wb'))
In [627]: np.load('test.pkl')
Out[627]: 
{'x': [array([2, 3, 4]), array([5, 6, 7])],
 'y': [array([1, 2, 3]), array([4, 5, 6])]}
In [629]: pickle.load(open('test.pkl','rb'))
Out[629]: 
{'x': [array([2, 3, 4]), array([5, 6, 7])],
 'y': [array([1, 2, 3]), array([4, 5, 6])]}

To write a csv from this dictionary I'd create a structured array with fields named 'x' and 'y'. But I think I'd have to concatenate the arrays, so I can produce a 1d array. The csv would then look something like:

x    y
2    1
3    4
....

Or if the subarrays are all the same length I might be able to produce

x    y
2 5  1 4
3 6  2 5
...

In any if you still want to go the csv route, you need to decide how represent the values as simple lines of columns.

Upvotes: 2

Related Questions