CODEWITHSUNDEEP
pythonpandastimedeltacumsum
SlimJim
SlimJim

Reputation: 169

Correcting Pandas Cumulative Sum on a Timedelta Column

I'm currently have a line of code I'm using to try to create a column that is based on a Cumulative sum of timedelta data between dates. How ever its not correctly performing the Cumulative sum everywhere, and I was also given a warning that my line of python code wont work in the future.

The original dataset is below:

ID       CREATION_DATE             TIMEDIFF              EDITNUMB       
8211    11/26/2019 13:00                                    1        
8211    1/3/2020 9:11      37 days 20:11:09.000000000       1       
8211    2/3/2020 14:52     31 days 05:40:57.000000000       1       
8211    3/27/2020 15:00    53 days 00:07:49.000000000       1       
8211    4/29/2020 12:07    32 days 21:07:23.000000000       1

Here is my line of python code:

df['RECUR'] = df.groupby(['ID']).TIMEDIFF.apply(lambda x: x.shift().fillna(1).cumsum())

Which produces the new column 'RECUR' that is not summing cumulatively correctly from the data in the 'TIMEDIFF' column:

ID       CREATION_DATE             TIMEDIFF              EDITNUMB          RECUR
8211    11/26/2019 13:00                                    1       0 days 00:00:01.000000000
8211    1/3/2020 9:11      37 days 20:11:09.000000000       1       0 days 00:00:02.000000000
8211    2/3/2020 14:52     31 days 05:40:57.000000000       1       37 days 20:11:11.000000000
8211    3/27/2020 15:00    53 days 00:07:49.000000000       1       69 days 01:52:08.000000000
8211    4/29/2020 12:07    32 days 21:07:23.000000000       1       122 days 01:59:57.000000000

Which also produces this warning:

FutureWarning: Passing integers to fillna is deprecated, will raise a TypeError in a future version.  To retain the old behavior, pass pd.Timedelta(seconds=n) instead.

Any help on this will be greatly appreciated, the sum total should be 153 days starting from 11/26/19, and correctly displayed cumulatively in the 'RECUR' column.

Upvotes: 1

Views: 1261

Answers (2)

Quang Hoang
Quang Hoang

Reputation: 150815

You can do:

# transform('first') would also work
df['RECUR'] = df['CREATION_DATE'] - df.groupby('ID').CREATION_DATE.transform('min')

Output:

     ID       CREATION_DATE         TIMEDIFF  EDITNUMB             RECUR
0  8211 2019-11-26 13:00:00              NaT         1   0 days 00:00:00
1  8211 2020-01-03 09:11:00 37 days 20:11:00         1  37 days 20:11:00
2  8211 2020-02-03 14:52:00 31 days 05:41:00         1  69 days 01:52:00
3  8211 2020-03-27 15:00:00 53 days 00:08:00         1 122 days 02:00:00
4  8211 2020-04-29 12:07:00 32 days 21:07:00         1 154 days 23:07:00

Upvotes: 2

jcaliz
jcaliz

Reputation: 4021

You can fillna with a timedelta of 0 seconds and do the cumsum

df['RECUR'] = df.groupby('ID').TIMEDIFF.apply(
    lambda x: x.fillna(pd.Timedelta(seconds=0)).cumsum())

df['RECUR']
# 0 0 days 00:00:00
# 1 37 days 20:11:09
# 2 69 days 01:52:06
# 3 122 days 01:59:55
# 4 154 days 23:07:18

Upvotes: 1

Related Questions