Filter on Pandas DataFrame with datetime columns raises error

Question

I'm setting up a DataFrame with two datetime columns like so:

range1 = Series(date_range('1/1/2011', periods=50, freq='D'))
range2 = Series(date_range('2/5/2011', periods=50, freq='D'))
df1    = DataFrame({'a': rng1, 'b': rng2}, dtype='datetime64[D]')

Oddly, asking the dtypes of df1 gives me:

In [71]: df1.dtypes
Out[71]:
a    datetime64[ns]
b    datetime64[ns]

What is worse, when I try to filter the dataframe like so:

In [62]:

d = datetime(2011,1,14)
df1[df1 > d]

I get an error:

TypeError                                 Traceback (most recent call last)
<ipython-input-62-50b4b9735157> in <module>()
      1 d = datetime(2011,1,14)
----> 2 df1[df1 > d]

/usr/local/lib/python2.7/site-packages/pandas/core/frame.pyc in f(self, other)
    313             return self._combine_series_infer(other, func)
    314         else:
--> 315             return self._combine_const(other, func)
    316 
    317     f.__name__ = name

/usr/local/lib/python2.7/site-packages/pandas/core/frame.pyc in _combine_const(self, other, func)
   3677             return self
   3678 
-> 3679         result_values = func(self.values, other)
   3680 
   3681         if not isinstance(result_values, np.ndarray):

TypeError: can't compare datetime.datetime to long

Could anyone tell me why this is happening? I am using numpy 1.7 and pandas 0.10.1.

Answer 1

only datetime64[ns] dtypes are supported, try w/o the dtype

In [9]: df1    = DataFrame({'a': range1, 'b' : range2})

In [10]: df1
In [15]: df1.head()
Out[15]: 
                    a                   b
0 2011-01-01 00:00:00 2011-02-05 00:00:00
1 2011-01-02 00:00:00 2011-02-06 00:00:00
2 2011-01-03 00:00:00 2011-02-07 00:00:00
3 2011-01-04 00:00:00 2011-02-08 00:00:00
4 2011-01-05 00:00:00 2011-02-09 00:00:00

In [16]: df1[df1.a>datetime.datetime(2011,1,14)].head()
Out[16]: 
                     a                   b
14 2011-01-15 00:00:00 2011-02-19 00:00:00
15 2011-01-16 00:00:00 2011-02-20 00:00:00
16 2011-01-17 00:00:00 2011-02-21 00:00:00
17 2011-01-18 00:00:00 2011-02-22 00:00:00
18 2011-01-19 00:00:00 2011-02-23 00:00:00

FYI, after this: https://github.com/pydata/pandas/issues/3311 is merged in, then the operation the OP indicated, a where returns this:

In [15]: df1[df1>datetime.datetime(2011,1,14)].head(20)
Out[15]: 
                     a                   b
0                  NaT 2011-02-05 00:00:00
1                  NaT 2011-02-06 00:00:00
2                  NaT 2011-02-07 00:00:00
3                  NaT 2011-02-08 00:00:00
4                  NaT 2011-02-09 00:00:00
5                  NaT 2011-02-10 00:00:00
6                  NaT 2011-02-11 00:00:00
7                  NaT 2011-02-12 00:00:00
8                  NaT 2011-02-13 00:00:00
9                  NaT 2011-02-14 00:00:00
10                 NaT 2011-02-15 00:00:00
11                 NaT 2011-02-16 00:00:00
12                 NaT 2011-02-17 00:00:00
13                 NaT 2011-02-18 00:00:00
14 2011-01-15 00:00:00 2011-02-19 00:00:00
15 2011-01-16 00:00:00 2011-02-20 00:00:00
16 2011-01-17 00:00:00 2011-02-21 00:00:00
17 2011-01-18 00:00:00 2011-02-22 00:00:00
18 2011-01-19 00:00:00 2011-02-23 00:00:00
19 2011-01-20 00:00:00 2011-02-24 00:00:00