linear least squares fit

One of the most common uses of least squares fitting is fitting a straight line todata. Whilst, in general, it is difficult to determine the curve which best fitsthe data, in this case there is a relatively simple formula which can be used.

Theorem 1.

Suppose we have a data set $\mathrm{(}{x}_{\mathrm{1}}\mathrm{,}{y}_{\mathrm{1}}\mathrm{)}\mathrm{,}\mathrm{\dots }\mathrm{,}\mathrm{(}{x}_n\mathrm{,}{y}_n\mathrm{)}$. Then the straightline which best fits this set is given as

$$y=\frac{ns-pq}{nr-p^2}x+\frac{qr-ps}{nr-p^2}$$

where

$p$$={\displaystyle \sum _{k=1}^n}{x}_k$(1)

$q$$={\displaystyle \sum _{k=1}^n}{y}_k$(2)

$r$$={\displaystyle \sum _{k=1}^n}{x}_k^2$(3)

$s$$={\displaystyle \sum _{k=1}^n}{x}_k{y}_k$(4)

Proof.

Being the best fitting line means minimizing the merit function $M$, given as

$$M(a,b)=\sum _{k=0}^n{(a{x}_k+b-y_k)}^2$$

with respect to the parameters $a$ and $b$. Expanding the square, this can bewritten as

$$M(a,b)=r{a}^2+2pab+n{b}^2-2sa-2qb+t$$

where $p,q,r,s$ are as above and

$$t=\sum _{k=1}^n{y}_k^2.$$

This function $M$ is a quadratic polynomial; moreover, from its definition as asum of squares, it is clear that the highest order terms are positive definite,hence it has a minimum and all that remains is to find that minimum. To do this,we set the derivatives equal to zero to obtain the following equations:

$0={\displaystyle \frac{\partial M(a,b)}{\partial a}}$$=2ar+2pb-2s$(5)

$0={\displaystyle \frac{\partial M(a,b)}{\partial b}}$$=2pa+2nb-2q$(6)

These equations are easily solved to give

$a$$={\displaystyle \frac{ns-pq}{nr-p^2}}$(7)

$b$$={\displaystyle \frac{qr-ps}{nr-p^2}};$(8)

substituting in the equation $y=ax+b$ for a straight line, we obtainthe answer given above.∎

Because of the ease with which one can make a least squares fit of a line, thistechnique is often adapted to fitting other sorts of curves by making a changeof variables. Two common cases of this practice are power laws and exponentials.

Suppose that one wants to fit some data to a curve of the form $y=c{e}^{kx}$.Making a change of variable $y=e^u$ and defining $b=\log c$, the equation ofthe curve becomes $u=kx+b$. One can therefore fit the data set $(x_1,\log y_1),\mathrm{\dots }(x_n,\log y_n)$ to a straight line.

Suppose that one wants to fit some data to a curve of the form $y=c{x}^p$. Makinga change of variable $x=e^v$, $y=e^u$ and defining $b=\log c$, the equation ofthe curve becomes $u=pv+b$. One can therefore fit the data set $(\log x_1,\log y_1),\mathrm{\dots }(\log x_n,\log y_n)$ to a straight line.

Although convenient and common, this procedure can be a cheat because changing variablesand making a least squares fit of a line is not the same as making a least squares fitto a curve. The reason for this is that the merit functions are different and will not,in general have a minimum in the same place. However, if the data happen to approximatelylie on a power curve or an exponential, then the answer obtained by changing variablesand fitting will be an approximation to the correct answer. Depending on what one isdoing, this approximation may be good enough or one may use it as a starting point forsome algorithm to compute the correct minimum.