CODEWITHSUNDEEP
pythonpandas
zio
zio

Reputation: 2225

HDFStore error appending - "Cannot serialize the column"

I have a dataframe, df:

    datetime                      bid      ask     bidvolume  askvolume
0   2007-03-30 21:00:00.332000   1.9682   1.9678       4         0.8

Trying to append this to a new datastore. The datastore does not exist so I use the following to create and append the data;

store = pd.HDFStore(storePath,mode='w')
store.append('data',df)
store.close()

I get this error: on the store.append line.

TypeError: Cannot serialize the column [bid] because
its data contents are [floating] object dtype

How do I get the data to store properly?

Upvotes: 7

Views: 14107

Answers (1)

Phillip Cloud
Phillip Cloud

Reputation: 25652

Please note: the following method convert_objects() is now deprecated and may not work Call DataFrame.convert_objects():

df = DataFrame(randn(10, 1), dtype=object).convert_objects()
df.to_hdf('/tmp/blah.h5', 'df', append=True)

It might be worth checking to see if you can get your data in the correct format before you start saving to HDF5. For example, wherever df is created, convert the objects there, instead of converting them when you save. In general, operations in pandas will be very cumbersome with a Series of floats with a dtype of object. Your life will be much easier if you convert your object arrays (where possible) as soon as you need to do anything with them.

Upvotes: 5

Related Questions