CODEWITHSUNDEEP
pythonpandastimedelta
Daniel Llewellyn
Daniel Llewellyn

Reputation: 11

Sum overflow TimeDeltas in Python Pandas

While trying to sum across timedeltas in pandas, it seems to work for a slice but not the whole column. 

>> d.ix[0:100, 'VOID-DAYS'].sum()
Timedelta('2113 days 00:00:00')

>> d['VOID-DAYS'].sum()


ValueError: overflow in timedelta operation

Upvotes: 1

Views: 3696

Answers (1)

unutbu
unutbu

Reputation: 879561

If VOID-DAYS represents an integer number of days, convert the Timedeltas into integers:

df['VOID-DAYS'] = df['VOID-DAYS'].dt.days

import numpy as np
import pandas as pd
df = pd.DataFrame({'VOID-DAYS': pd.to_timedelta(np.ones((106752,)), unit='D')})
try:
    print(df['VOID-DAYS'].sum())
except ValueError as err:
    print(err)
    # overflow in timedelta operation


df['VOID-DAYS'] = df['VOID-DAYS'].dt.days
print(df['VOID-DAYS'].sum())
# 106752

If the Timedeltas include seconds or smaller units, then use 

df['VOID-DAYS'] = df['VOID-DAYS'].dt.total_seconds()

to convert the value to a float.

Pandas Timedeltas (Series and TimedeltaIndexes) store all timedeltas as ints compatible with NumPy's timedelta64[ns] dtype. This dtype uses 8-byte ints to store the timedelta in nanoseconds.

The largest number of days representable in this format is 

In [73]: int(float(np.iinfo(np.int64).max) / (10**9 * 3600 * 24))
Out[73]: 106751

Which is why 

In [74]: pd.Series(pd.to_timedelta(np.ones((106752,)), unit='D')).sum()
ValueError: overflow in timedelta operation

raises a ValueError, but

In [75]: pd.Series(pd.to_timedelta(np.ones((106751,)), unit='D')).sum()
Out[75]: Timedelta('106751 days 00:00:00')

does not.

Upvotes: 8

Related Questions