CODEWITHSUNDEEP
pythonnumpypandastimestamp
Amelio Vazquez-Reina
Amelio Vazquez-Reina

Reputation: 96300

Unable to apply methods on timestamps using Series built-ins

On the following series:

0    1411161507178
1    1411138436009
2    1411123732180
3    1411167606146
4    1411124780140
5    1411159331327
6    1411131745474
7    1411151831454
8    1411152487758
9    1411137160544
Name: my_series, dtype: int64

This command (convert to timestamp, localize and convert to EST) works:

pd.to_datetime(my_series, unit='ms').apply(lambda x: x.tz_localize('UTC').tz_convert('US/Eastern'))

but this one fails:

pd.to_datetime(my_series, unit='ms').tz_localize('UTC').tz_convert('US/Eastern')

with:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-58187a4b60f8> in <module>()
----> 1 lua = pd.to_datetime(df[column], unit='ms').tz_localize('UTC').tz_convert('US/Eastern')

/Users/josh/anaconda/envs/py34/lib/python3.4/site-packages/pandas/core/generic.py in tz_localize(self, tz, axis, copy, infer_dst)
   3492                 ax_name = self._get_axis_name(axis)
   3493                 raise TypeError('%s is not a valid DatetimeIndex or PeriodIndex' %
-> 3494                                 ax_name)
   3495             else:
   3496                 ax = DatetimeIndex([],tz=tz)

TypeError: index is not a valid DatetimeIndex or PeriodIndex

and so does this one:

my_series.tz_localize('UTC').tz_convert('US/Eastern')

with: 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-0a7cb1e94e1e> in <module>()
----> 1 lua = df[column].tz_localize('UTC').tz_convert('US/Eastern')

/Users/josh/anaconda/envs/py34/lib/python3.4/site-packages/pandas/core/generic.py in tz_localize(self, tz, axis, copy, infer_dst)
   3492                 ax_name = self._get_axis_name(axis)
   3493                 raise TypeError('%s is not a valid DatetimeIndex or PeriodIndex' %
-> 3494                                 ax_name)
   3495             else:
   3496                 ax = DatetimeIndex([],tz=tz)

TypeError: index is not a valid DatetimeIndex or PeriodIndex

As far as I understand, the second approach above (the first one that fails) should work. Why does it fail?

Upvotes: 59

Views: 42319

Answers (3)

mocobk
mocobk

Reputation: 51

this work fine

pd.to_datetime(my_series,unit='ms', utc=True).dt.tz_convert('US/Eastern')

Upvotes: 4

John Zwinck
John Zwinck

Reputation: 249303

As Jeff's answer mentions, tz_localize() and tz_convert() act on the index, not the data. This was a huge surprise to me too.

Since Jeff's answer was written, Pandas 0.15 added a new Series.dt accessor that helps your use case. You can now do this:

pd.to_datetime(my_series, unit='ms').dt.tz_localize('UTC').dt.tz_convert('US/Eastern')

Upvotes: 125

Jeff
Jeff

Reputation: 128988

tz_localize/tz_convert act on the INDEX of the object, not on the values. Easiest to simply turn it into an index then localize and convert. If you then want a Series back you can use to_series()

In [47]: pd.DatetimeIndex(pd.to_datetime(s,unit='ms')).tz_localize('UTC').tz_convert('US/Eastern')
Out[47]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2014-09-19 17:18:27.178000-04:00, ..., 2014-09-19 10:32:40.544000-04:00]
Length: 10, Freq: None, Timezone: US/Eastern

Upvotes: 35

Related Questions