how to store a pandas Panel instance?

Question

There are numerous cases I need to output a light sized pandas Panel instance to the disk for later use. Currently I store it as pickle object as the pandas.read_pickle could directly retrieve it as a Panel object. But there are two pitfalls in doing this: First, I have to always note in the file name that this is is Panel object, otherwise I may forget. Second, does this has any risk in the future? For example, the future versions of pandas might not support this and I may lose the access to the data as a direct Panel? What are the other alternative ways to safely store a Panel object while it will be still easy to read it again directly as a Panel? I could also write panel to excel format but I have to read the member Dataframes one by one and combine them again into Panel when I get the data back.

Answer 1

you can use HDF as a storage.

Demo:

Let's generate a Panel with financial data:

import pandas as pd
import pandas_datareader.data as wb

stocks = ['AAPL', 'GOOG', 'FB']
p = wb.DataReader(stocks, 'yahoo', '2016-01-01')

now we have the following Panel:

In [10]: p.axes
Out[10]:
[Index(['Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object'),
 DatetimeIndex(['2016-01-04', '2016-01-05', '2016-01-06', '2016-01-07', '2016-01-08', '2016-01-11', '2016-01-12', '2016-01-13', '2016-01-14',
                '2016-01-15',
                ...
                '2017-02-02', '2017-02-03', '2017-02-06', '2017-02-07', '2017-02-08', '2017-02-09', '2017-02-10', '2017-02-13', '2017-02-14',
                '2017-02-15'],
               dtype='datetime64[ns]', name='Date', length=283, freq=None),
 Index(['AAPL', 'FB', 'GOOG'], dtype='object')]

Saving it to HDF5 file:

In [12]: p.to_hdf('c:/temp/panel.h5', 'p', format='t')

Check:

In [13]: store = pd.HDFStore('c:/temp/panel.h5')

In [14]: store
Out[14]:
<class 'pandas.io.pytables.HDFStore'>
File path: c:/temp/panel.h5
/p            wide_table   (typ->appendable,nrows->849,ncols->6,indexers->[major_axis,minor_axis])

In [15]: store.get_storer('p')
Out[15]: wide_table   (typ->appendable,nrows->849,ncols->6,indexers->[major_axis,minor_axis])

In [16]: store.get_storer('p').table
Out[16]:
/p/table (Table(849,)) ''
  description := {
  "major_axis": Int64Col(shape=(), dflt=0, pos=0),
  "minor_axis": StringCol(itemsize=4, shape=(), dflt=b'', pos=1),
  "values_block_0": Float64Col(shape=(6,), dflt=0.0, pos=2)}
  byteorder := 'little'
  chunkshape := (1092,)
  autoindex := True
  colindexes := {
    "major_axis": Index(6, medium, shuffle, zlib(1)).is_csi=False,
    "minor_axis": Index(6, medium, shuffle, zlib(1)).is_csi=False}

In [17]: x = store['p']

In [18]: x
Out[18]:
<class 'pandas.core.panel.Panel'>
Dimensions: 6 (items) x 283 (major_axis) x 3 (minor_axis)
Items axis: Open to Adj Close
Major_axis axis: 2016-01-04 00:00:00 to 2017-02-15 00:00:00
Minor_axis axis: AAPL to GOOG

In [20]: x.loc[:,:,'GOOG']
Out[20]:
                  Open        High         Low       Close     Volume   Adj Close
2016-01-04  743.000000  744.059998  731.257996  741.840027  3272800.0  741.840027
2016-01-05  746.450012  752.000000  738.640015  742.580017  1950700.0  742.580017
2016-01-06  730.000000  747.179993  728.919983  743.619995  1947000.0  743.619995
2016-01-07  730.309998  738.500000  719.059998  726.390015  2963700.0  726.390015
2016-01-08  731.450012  733.229980  713.000000  714.469971  2450900.0  714.469971
2016-01-11  716.609985  718.854980  703.539978  716.030029  2090600.0  716.030029
2016-01-12  721.679993  728.750000  717.317017  726.070007  2024500.0  726.070007
2016-01-13  730.849976  734.739990  698.609985  700.559998  2501700.0  700.559998
2016-01-14  705.380005  721.924988  689.099976  714.719971  2225800.0  714.719971
2016-01-15  692.289978  706.739990  685.369995  694.450012  3592400.0  694.450012

...