CODEWITHSUNDEEP
pythonpandasdataframe
measure_theory
measure_theory

Reputation: 874

Pandas: filling missing values in time series forward using a formula

I have a time series of data in a DataFrame that has missing values at both the beginning and the end of the sample.

I'm trying to fill the missing values at the end by growing it forward using a simple AR(1) process.

For example,

X(t+1) - X(t) = 0.5*[X(t) - X(t-1)]

A = [np.nan, np.nan, 5.5, 5.7, 5.9, 6.1, 6.0, 5.9, np.nan, np.nan, np.nan]  

df = pd.DataFrame({'A':A}, index = pd.date_range(start = '2010', 
                                                 periods = len(A),
                                                 freq = "QS"))

              A
2010-01-01  5.5
2010-04-01  5.7
2010-07-01  5.9
2010-10-01  6.1
2011-01-01  6.0
2011-04-01  5.9
2011-07-01  NaN
2011-10-01  NaN
2012-01-01  NaN

What I want:

                 A
2010-01-01     NaN
2010-04-01     NaN
2010-07-01  5.5000
2010-10-01  5.7000
2011-01-01  5.9000
2011-04-01  6.1000
2011-07-01  6.0000
2011-10-01  5.9000
2012-01-01  5.8500
2012-04-01  5.8250
2012-07-01  5.8125

Grabbing the next entry in the series is relatively easy:

NEXT = 0.5*df.dropna().diff().iloc[-1] + df.dropna().iloc[-1]

But appending that to the DataFrame in a nice ways is giving me some trouble.

Upvotes: 0

Views: 597

Answers (1)

nandneo
nandneo

Reputation: 505

You can use the below code to do the operation:

A = [np.nan, np.nan, 5.5, 5.7, 5.9, 6.1, 6.0, 5.9, np.nan, np.nan, np.nan]

df = pd.DataFrame({'A': A}, index=pd.date_range(start='2010', periods=len(A), freq="QS"))

for id in df[df.A.isnull() == True].index:
    df.loc[id, 'A'] = 1.5 * df.A.shift().loc[id] - 0.5 * df.A.shift(2).loc[id]

#Output dataframe
                 A
2010-01-01     NaN
2010-04-01     NaN
2010-07-01  5.5000
2010-10-01  5.7000
2011-01-01  5.9000
2011-04-01  6.1000
2011-07-01  6.0000
2011-10-01  5.9000
2012-01-01  5.8500
2012-04-01  5.8250
2012-07-01  5.8125

Upvotes: 2

Related Questions