CODEWITHSUNDEEP
pandaspytables
fantabolous
fantabolous

Reputation: 22696

pandas HDFStore select rows by datetime index

I'm sure this is probably very simple but I can't figure out how to slice a pandas HDFStore table by its datetime index to get a specific range of rows.

I have a table that looks like this:

mdstore = pd.HDFStore(store.h5)
histTable = '/ES_USD20120615_MIDPOINT30s'
print(mdstore[histTable])
                         open      high       low     close  volume  WAP  \
date                                                                       
2011-12-04 23:00:00  1266.000  1266.000  1266.000  1266.000      -1   -1   
2011-12-04 23:00:30  1266.000  1272.375  1240.625  1240.875      -1   -1   
2011-12-04 23:01:00  1240.875  1242.250  1240.500  1242.125      -1   -1   
                   ...
[488000 rows x 7 columns]

For example I'd like to get the range from 2012-01-11 23:00:00 to 2012-01-12 22:30:00. If it were in a df I would just use datetimes to slice on the index, but I can't figure out how to do that directly from the store table so I don't have to load the whole thing into memory. I tried mdstore.select(histTable, where='index>20120111') and that worked in as much as I got everything on the 11th and 12th, but I couldn't see how to add a time in.

Upvotes: 4

Views: 2463

Answers (1)

Jeff
Jeff

Reputation: 128918

Example is here needs pandas >= 0.13.0

In [2]: df = DataFrame(np.random.randn(5),index=date_range('20130101 09:00:00',periods=5,freq='s'))

In [3]: df
Out[3]: 
                            0
2013-01-01 09:00:00 -0.110577
2013-01-01 09:00:01 -0.420989
2013-01-01 09:00:02  0.656626
2013-01-01 09:00:03 -0.350615
2013-01-01 09:00:04 -0.830469

[5 rows x 1 columns]

In [4]: df.to_hdf('test.h5','data',mode='w',format='table')

Specify it as a quoted string

In [8]: pd.read_hdf('test.h5','data',where='index>"20130101 09:00:01" & index<"20130101 09:00:04"')
Out[8]: 
                            0
2013-01-01 09:00:02  0.656626
2013-01-01 09:00:03 -0.350615

[2 rows x 1 columns]

You can also specify it directly as a Timestamp

In [10]: pd.read_hdf('test.h5','data',where='index>Timestamp("20130101 09:00:01") & index<Timestamp("20130101 09:00:04")')
Out[10]: 
                            0
2013-01-01 09:00:02  0.656626
2013-01-01 09:00:03 -0.350615

[2 rows x 1 columns]

Upvotes: 8

Related Questions