CODEWITHSUNDEEP
pythonpandaslinear-regression
John John
John John

Reputation: 85

Finding the least squares linear regression for each row of a dataframe in python using pandas

I have the a dataframe:

(day/month/year)

df = pd.DataFrame({'Name': ['A', 'B', 'C'], 
                   'Date0': ['01/01/1999','01/06/1999','01/01/1979'], 'V0': [29,44,21],
                   'Date1': ['08/01/2000','07/01/2000','01/01/2000'],'V1': [35, 45, 47]})

I want to interpolate the age for each row to find 'V_10' which is the value at the date 10/08/1999 using a linear regression. For example in the first case I would get something like:

Slope   0.01609
Y-intercept     29.00    
df = pd.DataFrame({'Name': ['A', 'B', 'C'], 
                   'Date0': ['01/01/1999','01/06/1999','01/01/1979'], 'V0': [29,44,21],
                   'Date1': ['08/01/2000','07/01/2000','01/01/2000'],'V1': [35, 45, 47], 
                   'V_10':[32.57]})

I hope my calculations are correct.

And what if I want an exponential regression or worse a custom function I have?

Upvotes: 1

Views: 2146

Answers (1)

rhozzy
rhozzy

Reputation: 352

I'm not sure if this is what you're after, but for a linear interpolation you could do the following:

import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline

df = pd.DataFrame({'Name': ['A', 'B', 'C'], 
               'Date0': ['01/01/1999','01/06/1999','01/01/1979'], 'V0': [29,44,21],
               'Date1': ['08/01/2000','07/01/2000','01/01/2000'],'V1': [35, 45, 47]})
df['Target'] = pd.to_datetime('10/08/1999')
df['Date0'] = pd.to_datetime(df['Date0'])
df['Date1'] = pd.to_datetime(df['Date1'])
df['Target'] = pd.to_datetime(df['Target'])

def regress(xs, ys, newx, reference=pd.to_datetime('1/1/1900'), retype='linear', fit_intercept=True, degree=None):
    xs = [(x - reference).days for x in xs]
    xs = np.array(xs).reshape(-1,1)
    ys = np.array(ys)
    if retype == 'linear':
        lm = LinearRegression(fit_intercept=fit_intercept)
    elif retype == 'polynomial':
        lm = Pipeline([('poly', PolynomialFeatures(degree=degree)),
                   ('linear', LinearRegression(fit_intercept=fit_intercept))])
    else:
        return print('Need to specify other regression type.')
    lm.fit(xs,ys)
    return lm.predict(np.array((newx - reference).days).reshape(-1, 1))[0]

# Linear regression example
df['V10'] = df.apply(lambda x: regress([x.Date0,x.Date1], [x.V0,x.V1], x.Target, retype='linear'), axis=1)
# 2nd-degree polynomial regression example
df['V11']=df.apply(lambda x: regress([x.Date0,x.Date1], [x.V0,x.V1], x.Target, retype='polynomial', degree=2), axis=1)

Upvotes: 1

Related Questions