CODEWITHSUNDEEP
pythonpandas
bigbug
bigbug

Reputation: 59544

Pandas datetime convert & filter issue

I am using Pandas 0.8.1 to fetch Yahoo stock price

from datetime import datetime
from pandas.io.data import DataReader

stk_price = DataReader('600809.ss', 'yahoo', datetime(2006,1,1), datetime(2012,12,31)).reset_index()

>>> stk_price.Date
0    2010-01-04 00:00:00
1    2010-01-05 00:00:00
2    2010-01-06 00:00:00
3    2010-01-07 00:00:00
4    2010-01-08 00:00:00

And I want to convert the Date to string by:

>>>stk_price.Date.astype('|S10')
0     1970-01-15
1     1970-01-15
2     1970-01-15
3     1970-01-15
4     1970-01-15
5     1970-01-15

Why it shows "1970-01-15" instead of "2010-01-04" etc ? How to fix it?

And if I have a 

DATE_LIST = [
 u'20090331', u'20090630', u'20090930', u'20091231', \
 u'20100331', u'20100630', u'20100930', u'20101231', \
 u'20110331', u'20110630', u'20110930', u'20111231', \
 u'20120331', u'20120630', u'20120930', u'20121231'
 ]

I just try to filter the rows of stk_price whose Date column is in DATE_LIST by below: 

from datetime import datetime
from pandas.io.data import DataReader
import numpy as np

DATE_LIST = [
 u'20090331', u'20090630', u'20090930', u'20091231', \
 u'20100331', u'20100630', u'20100930', u'20101231', \
 u'20110331', u'20110630', u'20110930', u'20111231', \
 u'20120331', u'20120630', u'20120930', u'20121231'
 ]

DATE_ARRAY = np.array(DATE_LIST,dtype='datetime64[us]')
stk_price = DataReader('600809.ss', 'yahoo', datetime(2006,1,1), datetime(2012,12,31)).reset_index()
rst = stk_price[stk_price.Date.isin(DATE_ARRAY)].Close

but the rst is empty .

How to fix it or any Pandas function can filter the result ?

Upvotes: 0

Views: 4099

Answers (2)

root
root

Reputation: 80406

use df.Date=df.Date.apply(lambda x:x.date()) to convert the datetime column

>>> df=DataFrame({'Date':[datetime(2006,1,1)]})
>>> df
                  Date
0  2006-01-01 00:00:00
>>> df.Date=df.Date.apply(lambda x:x.date())
>>> df
         Date
0  2006-01-01

EDIT:

to format DATE_LIST

dates=[datetime.datetime.strptime(i, "%Y%m%d").date() for i in DATE_LIST]

and finally use isin to filter out the rows that are in DATE_LIST:

df[df['Dates'].isin(dates)]

in one go(given that you have stk_price dataframe that has a datecolumn called Date):

import datetime

stk_price = DataReader('600809.ss', 'yahoo', datetime(2006,1,1), datetime(2012,12,31)).reset_index()
stk_price.Date=stk_price.Date.apply(lambda x:x.date())
dates=[datetime.datetime.strptime(i, "%Y%m%d").date() for i in DATE_LIST]
stk_price[stk_price['Dates'].isin(dates)]

Upvotes: 2

Wouter Overmeire
Wouter Overmeire

Reputation: 69236

Instead of converting stk_price dates to strings (which shows the wrong dates due to a numpy bug, see also github issue #1802) you can also convert the DATE_LIST to timestamps and use these directly to index stk_price. Below an example on frame containing random data. 

In [16]: DATE_LIST = [
 u'20090331', u'20090630', u'20090930', u'20091231', \
 u'20100331', u'20100630', u'20100930', u'20101231', \
 u'20110331', u'20110630', u'20110930', u'20111231', \
 u'20120331', u'20120630', u'20120930', u'20121231'
 ]

In [17]: timestamps = [pd.Timestamp(date) for date in DATE_LIST]

In [18]: df = pd.DataFrame(np.random.randn(100,3), index=pd.date_range('20090331', periods=100))

In [19]: df.ix[timestamps].dropna()
Out[19]:
                   0         1         2
2009-03-31  0.520235  1.158889 -0.310227
2009-06-30  1.036449  0.528931 -0.083166

Upvotes: 1

Related Questions