CODEWITHSUNDEEP
pythonpandaspytables
fantabolous
fantabolous

Reputation: 22706

save currencies to pandas store with precision (as Decimal?)

In pandas I work a lot with currencies. Up to this point I've been using the default floats, but dealing with the lack of precision is annoying and error prone. I'm trying to switch over to using Decimal for some pieces, which while it likely makes calculations a lot slower, is precise. However when I try to save to a pandas store (e.g. hdf5store via pytables) I get: TypeError: Cannot serialize the column [o] because its data contents are [mixed] object dtype

Here's a short sample of what I'm trying to do:

import pandas as pd
from decimal import Decimal
teststore = pd.HDFStore('teststore.h5')
df = pd.DataFrame(data={'o':[Decimal('5.1')]})
teststore['test'] = df

.. which raises the exception above. df.convert_objects(convert_numeric=True) doesn't help.

Is there a way to save Decimal to a pandas store, and if not, is there a recommended way of precisely storing currencies in a pandas store?

I'm using python 2.7.8, pandas 0.14.1, and pytables 3.1.1.

Upvotes: 1

Views: 793

Answers (1)

Jeff
Jeff

Reputation: 129018

Works on 0.15.0. Though it is essentially pickled as its an actual python object, so you get almost no benefit from using HDF5.

In [46]: from decimal import Decimal

In [47]: teststore = pd.HDFStore('teststore.h5')

In [48]: df = pd.DataFrame(data={'o':[Decimal('5.1')]})

In [49]: teststore['test'] = df
pandas/io/pytables.py:2487: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->mixed,key->block0_values] [items->['o']]

  warnings.warn(ws, PerformanceWarning)

As an FYI, generally float64 has 14-16 digits of precision, so not sure why you are not using them (you may need to change the display printing precision to see it).

In [50]: In [34]: pd.set_option('precision',16)

In [51]: In [35]: s = Series([0.0000000000001,0.000000000000002])

In [52]: s+s
Out[52]: 
0    0.000000000000200
1    0.000000000000004
dtype: float64

Upvotes: 1

Related Questions