Least Squares Method to fit parameters

Question

I am asked to use the least squares method to fit the parameters α and β in y = α*exp(-β*x),

given the points:

x = [1 2 3 4 5 6 7]
y = [9 6 4 2 4 6 9]

I am having trouble determining what my matrix should look like. I know I should take the natural logarithm of both sides of the function in order to get rid of the exponential, and also obtain the natural logarithm of the y-values, which are:

ln_y = [2.19 1.79 1.39 0.69 1.39 1.79 2.19]

However what should my matrix look like, because what I am left with is ln(y) = ln(α) - β*x?

So the -β column consists of ones and the x column will be my x values, but what should the α column contain?

This is what I assume I should get:

A = [1 1 1 1 1 1 1; 1 2 3 4 5 6 7]

Am I thinking correctly?

Answer 1

In this example, deriving the least squares estimator is a good idea. The other answers take this approach.

There is quick and dirty approach that is flexible and handy.

Just to it numerically. You can use fminsearch to get the job done.

% MATLAB R2019a
x = [1 2 3 4 5 6 7];
y = [9 6 4 2 4 6 9];

% Create anonymous function (your supposed model to fit)
fh =@(params) params(1).*exp(-params(2).*x);

% Create anonymous function for Least Squares Error with single input
SSEh =@(params) sum((fh(params)-y).^2);     % Sum of Squared Error (SSE)

p0 = [1 0.5];                   % Initial guess for [alpha, beta]
[p, SSE] = fminsearch(SSEh,p0);
alpha = p(1);           % 5.7143
beta = p(2);            % 1.2366e-08  (AKA zero)

It is always a good idea to plot the results as a sanity check (I screw up often and this saves me time and time again).

yhath=@(params,xval) params(1).*exp(-params(2).*xval);

Xrng = min(x)-1:.2:max(x)+1;
figure, hold on, box on
plot(Xrng,p(1).*exp(-p(2).*Xrng),'r--','DisplayName','Fit')
plot(x,y,'ks','DisplayName','Data')
legend('show')

A Note on Nonlinearity:
This works fine with linear models due to convexity. If your error function is nonlinear but convex, as Sum of Squared Error (SSE), then this also returns the global optimum.

Note that a non-convex function would require multiple start points to attempt to capture many local optima, then taking the best one would still carry no guarantees of optimality. Adding constraints to the solution would require penalty functions or switching to the constrained solver since fminsearch solves the unconstrained problem (unless you penalize it properly).

Easy to Modify:
It is easy to modify the model and the error function. For example, if you wanted to minimize the sum of the absolute error instead, it is straightforward using abs.

% Create anonymous function for Least Absolute Error with single input
SAEh =@(params) sum(abs(fh(params)-y));     % Sum of Absolute Error

Answer 2

You almost had it. The second row should be -x.

x = [1 2 3 4 5 6 7]
y = [9 6 4 2 4 6 9]

logy = log(y)

n = length(x);
A = [ones(1,n); -x]

c = logy/A; %Solve for coefficients

alpha = exp(c(1))
beta = c(2);

Answer 3

The first thing we can do is to take the natural logarithm ln (log in Matlab)) on both sides of the equation:

y = α * e^(-β * x)

becomes:

ln(y) = ln(α * e^(-β * x))
// Law of logarithms
ln(x * y) = ln(x) + ln(y) 

// thus:
ln(y) = ln(α) + ln(e^(-β * x))
Simplifying:
ln(y) = -β * x + ln(α)

Now we have ln(y) as a linear function of x and the problem reduces to finding the linear regression in the least square sense. Let's define lny = log(y), and A = ln(α) and we can rewrite the problem as

lny = -β * x + A

Where

x = [1 2 3 4 5 6 7]
lny = [2.19 1.79 1.39 0.69 1.39 1.79 2.19]

For each x_i in x we can evaluate lny as follows (rewritten in ascending power of x):

lny(x1) = A - β * x1
lny(x2) = A - β * x2
...
lny(xn) = A - β * xn

In matrix form

LNY = X * [A β]'
Or,
X * [A β]' = LNY
// let Coefs = [A β]'
Coefs = X^-1 * LNY

In Matlab

x = [1 2 3 4 5 6 7];
y = [9 6 4 2 4 6 9];
lny = log(y);
X = [ones(length(y), 1), -x']; % design matrix
coefs = X\lny'
% A = coefs(1) and β = coefs(2)
% ln(α) = A thus α = exp(A)
alpha = exp(coefs(1));
beta = coefs(2)