CODEWITHSUNDEEP
regressionoctaveoctave-gui
beginner
beginner

Reputation: 183

The problem with the dimensions of a matrix when computing the least squares method

My previous topic in this area.

Problem in solving algorithm polynomial regression,least squares method in Octave

I decided not to change the main questions, but to create a new question for each problem.

This time, the problem is as follows.I worked with least square method in polynomial regression.

Differentiating this function with respect to the vector of parameters b and equating the derivatives to zero, we obtain the system of equations (in matrix form).This formula.

(X^{T}X)b=X^{T}y.

That is, if I want to express b from here, I need to do what I did.

b=X^{T}y/(X^{T}X)=> b=X'*y1/(X'*X)

Code:

#Generating random values ​​of experimental data
x=0:0.1:5;
y=3*x+2;
y1=y+randn(size(y));
k=5;#Polynomial degree
X=[x' ones(length(x),1)];
b=X'*y1/(X'*X); Error: operator *: nonconformant arguments (op1 is 2x51, op2 is 1x51)\

Yes, the dimensions of the arrays X and y1 do not coincide. Attached screenshots will show everything.In this screenshot variables X,y1 and X'(transposed).

enter image description here

Then I decided to select 1 column from the array X and multiply it by y1.

enter image description here

If you look at the screenshots, you will see that now the transposed X' matches the y1 dimensions. That is, there should be no error, but it still exists.

Upvotes: 0

Views: 120

Answers (1)

Ander Biguri
Ander Biguri

Reputation: 35525

You math is wrong. You can not just randomly divide where you want when doing matrix operations.

The least square solution is

b=(X'*X)^-1 * X' * y

Also your definition of y is wrong, or your sized don't fit.

x=0:0.1:5;
y=(3*x+2)'; %transpose!
y1=y+randn(size(y));
X=[x' ones(length(x),1)];

The inverse of a matrix is something you should never take numerically. Luckily there is something called the Moore-Penrose pseudoinverse, which ensures that when used, you will get the least squares solution to your original problem. So lets use that.

b=pinv(X'*X)*X'*y;
b_with_noise=pinv(X'*X)*X'*y1;

Note that the following is basically zero to numerical precission, which mean it is correct.

sum(X*b-y)

ans =

  -5.8176e-14

Now, in MATLAB/octave you can do this much easier as :

b=X\y;
b_with_noise=X\y1;

MATLAB will find the best algorithm to compute the least squares solution. mldivide has more information (octave docs are a bit more lacking, but should work in the same way).

Upvotes: 2

Related Questions