CODEWITHSUNDEEP
pythonperformancepandas
coffeequant
coffeequant

Reputation: 615

pandas - extremely extremely slow

I am trying to do a df.apply on date objects but it's too too slow!!

My prun output gives....

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1999   14.563    0.007   14.563    0.007 {pandas.tslib.array_to_timedelta64}
 13998    0.103    0.000   15.221    0.001 series.py:126(__init__)
  9999    0.093    0.000    0.093    0.000 {method 'reduce' of 'numpy.ufunc' objects}
272012    0.093    0.000    0.125    0.000 {isinstance}
  5997    0.089    0.000    0.196    0.000 common.py:199(_isnull_ndarraylike)

So basically it's 14 seconds for a 2000 length array. My actual array size is > 100,000 which translates to a run time of > 15 minutes or maybe more.

It's stupid of pandas to call this function "pandas.tslib.array_to_timedelta64" which is the bottleneck? I really don't understand why this function call is necessary??? Both the operators in subtraction are of same data types. I explicity converted them beforehand using pd.to_datetime() method. And no this conversion time is not included in this calculation.

So in all you can understand my frustration at this pathetic code!!!

actual code looks like this

 df  = pd.DataFrame(bet_endtimes)

def testing():
    close_indices = df.apply(lambda x: np.argmin(np.abs(currentdata['date'] - x[0])),axis=1)
    print close_indices

 %prun testing()

Upvotes: 1

Views: 10793

Answers (1)

Jeff
Jeff

Reputation: 129068

I'd recommend consulting the documentation: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#time-deltas Its also very helpful to include sample data so I don't have to guess what you are doing.

Using apply is always the last operation to try. Vectorized methods are much faster.

In [55]: pd.set_option('max_rows',10)

In [56]: df = DataFrame(dict(A = pd.date_range('20130101',periods=100000, freq='s')))

In [57]: df
Out[57]: 
                        A
0     2013-01-01 00:00:00
1     2013-01-01 00:00:01
2     2013-01-01 00:00:02
3     2013-01-01 00:00:03
4     2013-01-01 00:00:04
...                   ...
99995 2013-01-02 03:46:35
99996 2013-01-02 03:46:36
99997 2013-01-02 03:46:37
99998 2013-01-02 03:46:38
99999 2013-01-02 03:46:39

[100000 rows x 1 columns]

In [58]:  (df['A']-df.loc[10,'A']).abs()
Out[58]: 
0   00:00:10
1   00:00:09
2   00:00:08
...
99997   1 days, 03:46:27
99998   1 days, 03:46:28
99999   1 days, 03:46:29
Name: A, Length: 100000, dtype: timedelta64[ns]

In [59]: %timeit  (df['A']-df.loc[10,'A']).abs()
1000 loops, best of 3: 1.47 ms per loop

When you contribute to pandas, you can name methods.

It's stupid of pandas to call this function "pandas.tslib.array_to_timedelta64" which is the bottleneck? time is not included in this calculation.

Upvotes: 8

Related Questions