CODEWITHSUNDEEP
matlabregression
math11
math11

Reputation: 537

Ridge coefficients larger than least squares

I've been using Matlab to compute coefficients for a model using both least squares and ridge. I was pretty sure that all my coding has been correct.

But for one dataset, (Boston housing), the ridge coefficients are larger than the least squares coefficients. Is this actually possible? What does it means? Or have I made a coding error?.....

Upvotes: 0

Views: 226

Answers (1)

DanielTheRocketMan
DanielTheRocketMan

Reputation: 3249

It seems that it might not be a problem at all...

1) In the typical least squares problem, you have to choose beta vector that minimizes

||y-X*beta||^2

2) Another associated problem (known as Lasso problem) is to find beta vector that minimizes

||y-X*beta||^2 + lambda*||beta||

3) Finally, in the ridge regression, your problem is to find beta vector that minimizes

||y-X*beta||^2 + lambda*||beta||^2

Note that in problem (2) above, it is clear that you are specifically penalizing the size of the [beta_i]s.

On the other hand, in problem (3) above, you are penalizing the differences in the sizes of the betas_i. I mean if you have in the vector beta, small beta_i s and large beta_i s, your cost is still going to be large. Imagine that the vector beta=[0.1;0.0001] in the problem (1). While to reduce "proportionally" both beta_is in problem (2) seems to be a good solution, the same does not happen in problem (3), where the best is to increase a little the size of beta_2=0.0001 in order to reduce more the size of beta_1=0.1.

Therefore, if your matlab solution of problem (3) presents beta_i s with sizes more similar, it seems that you are doing well.

I hope I help, but I never run this kind of regression before and I dont have the matlab here as well.

Upvotes: 1

Related Questions