CODEWITHSUNDEEP
machine-learningnormalizationrapidminer
Johnny
Johnny

Reputation: 33

Effect of Standardization in Linear Regression: Machine Learning

As part of my assignment, I am working on couple of datasets, and finding their training errors with linear Regression. I was wondering whether the standardization has any effect on the training error or not? My correlation, and RMSE is coming out to be equal for datasets before and after the standardization.

Thanks,

Upvotes: 0

Views: 782

Answers (1)

lejlot
lejlot

Reputation: 66795

It is easy to show that for linear regression it does not matter if you just transform input data through scaling (by a; the same applies for translation, meaning that any transformation of the form X' = aX + b for real a != 0,b have the same property). 

X' = aX

w = (X^TX)X^Ty
w' = (aX^TaX)^-1 aX^Ty
w' = 1/a w

Thus

X^Tw = 1/a aX^T w = aX^T 1/a w = X'^Tw'^T

Consequently the projection, where the error is computed is exactly the same before and after scaling, so any type of loss function (independent on x) yields the exact same results.

However, if you scale output variable, then errors will change. Furthermore, if you standarize your dataset in more complex way then by just multiplying by a number (for example - by whitening or by nearly any rotation) then your results will depend on the preprocessing. If you use regularized linear regression (ridge regression) then even scaling the input data by a constant matters (as it changes the "meaning" of regularization parameter).

Upvotes: 5

Related Questions