CODEWITHSUNDEEP
pythonpandashdf5pytables
Jacques MALAPRADE
Jacques MALAPRADE

Reputation: 1013

Can't save pandas dataframe to HDF file

I have been trying for a while to save a pandas dataframe to an HDF5 file. I tried various different phrasings eg. df.to_hdf etc. but to no avail. I am running this in a python virtual environment see here. Even without the use of the VE it has the same error. The following script comes up with the error below:

''' This script reads in a pickles dictionary converts it to panda 
dataframe and then saves it to an hdf file. The arguments are the 
file names of the pickle files.
'''

import numpy as np
import pandas as pd
import pickle
import sys

# read in filename arguments
for fn in sys.argv[1:]:
    print 'converting file %s to hdf format...' % fn
    fl = open(fn, 'r')
    data = pickle.load(fl)
    fl.close()
    frame = pd.DataFrame(data)
    fnn = fn.split('.')[0]+'.h5'
    store = pd.HDFStore(fnn)
    store.put([fn.split('.')[0]], frame)
    store.close()
    frame = 0
    data = 0

Error is:

$ ./p_to_hdf.py LUT_*.p
converting file LUT_0.p to hdf format...
Traceback (most recent call last):
  File "./p_to_hdf.py", line 22, in <module>
    store = pd.HDFStore(fnn)
  File "/usr/lib/python2.7/site-packages/pandas/io/pytables.py", line 270, in __init__
    raise Exception('HDFStore requires PyTables')
Exception: HDFStore requires PyTables

pip list shows both pandas and tables are installed and the latest versions.

pandas (0.16.2)
tables (3.2.0)

Upvotes: 0

Views: 1940

Answers (1)

Jacques MALAPRADE
Jacques MALAPRADE

Reputation: 1013

The solution had noting to do with the code but how to source a virtual environment in python. The correct way is to use . venv/bin/activate instead of source ~/venv/bin/activate. Now which python shows the python installed under ~/venv/bin/python and the code runs correctly.

Upvotes: 1

Related Questions