$$

Linear regression

Problem

Consider the data:

i	xi	yi
1	1	1
2	2	1
3	3	2
4	4	2

Find a straight line

$f(x)=c_0+c_{1}x$ such that

$\sum _{i=1}^4(f(x_i)-y_i{)}^2$ is minimized.

Thought

For any given data set

$(x_1,y_1),\dots ,(x_N,y_N)$ and the straight line

$f(x)=c_0+c_{1}x$, the key observation here is that

$$\left[\begin{array}{c}f(x_1)\\ ⋮\\ f(x_N)\end{array}\right]=\left[\begin{array}{c}{c}_0+c_1{x}_1\\ ⋮\\ c_0+c_1{x}_N\end{array}\right]=c_0\left[\begin{array}{c}1\\ ⋮\\ 1\end{array}\right]+c_1\left[\begin{array}{c}{x}_1\\ ⋮\\ x_N\end{array}\right]=\left[\begin{array}{cc}1& x_1\\ ⋮& ⋮\\ 1& x_N\end{array}\right]\left[\begin{array}{c}{c}_0\\ c_1\end{array}\right].$$

With

$$A=\left[\begin{array}{cc}1& x_1\\ ⋮& ⋮\\ 1& x_N\end{array}\right],\mathbf{c}=\left[\begin{array}{c}{c}_0\\ c_1\end{array}\right],\text{ and }\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_N\end{array}\right],$$

we are looking for appropriate

$\mathbf{c}$ to minimize

$\Vert A\mathbf{c}-\mathbf{y}{\Vert }^2$. This is a least square problem, and we know that the answer is

$\mathbf{c}=(A^{\mathrm{\top }}A{)}^{-1}{A}^{\mathrm{\top }}\mathbf{y}$.

Sample answer

Let

$$A=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\\ 1& 4\end{array}\right]\text{ and }\mathbf{y}=\left[\begin{array}{c}1\\ 1\\ 2\\ 2\end{array}\right].$$

Then the answer is

$$\begin{array}{rl}\mathbf{c}& =(A^{\mathrm{\top }}A{)}^{-1}{A}^{\mathrm{\top }}\mathbf{y}\\ & ={\left[\begin{array}{cc}4& 10\\ 10& 30\end{array}\right]}^{-1}\left[\begin{array}{cccc}1& 1& 1& 1\\ 1& 2& 3& 4\end{array}\right]\left[\begin{array}{c}1\\ 1\\ 2\\ 2\end{array}\right]\\ & =\left[\begin{array}{c}0.5\\ 0.4\end{array}\right].\end{array}$$

Thus,

$f(x)=0.5+0.4x$ is the straight line that best describes the data, which is also known as the best fitting line .

This note can be found at Course website > Learning resources.