$$

Polynomial regression

Problem

Consider the data:

i	xi	yi
1	1	2
2	2	0
3	3	0
4	4	1

Find a polynomial

$f(x)=c_0+c_{1}x+c_2{x}^2$ such that

$\sum _{i=1}^4(f(x_i)-y_i{)}^2$ is minimized.

Thought

For any given data set

$(x_1,y_1),\dots ,(x_N,y_N)$ and the straight line

$f(x)=c_0+c_{1}x+c_2{x}^2$, the key observation here is that

$$\left[\begin{array}{c}f(x_1)\\ ⋮\\ f(x_N)\end{array}\right]=\left[\begin{array}{c}{c}_0+c_1{x}_1+c_2{x}_1^2\\ ⋮\\ c_0+c_1{x}_N+c_2{x}_N^2\end{array}\right]=\left[\begin{array}{ccc}1& x_1& x_1^2\\ ⋮& ⋮& ⋮\\ 1& x_N& x_N^2\end{array}\right]\left[\begin{array}{c}{c}_0\\ c_1\\ c_2\end{array}\right].$$

With

$$A=\left[\begin{array}{ccc}1& x_1& x_1^2\\ ⋮& ⋮& ⋮\\ 1& x_N& x_N^2\end{array}\right],\mathbf{c}=\left[\begin{array}{c}{c}_0\\ c_1\\ c_2\end{array}\right],\text{ and }\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_N\end{array}\right],$$

we are looking for appropriate

$\mathbf{c}$ to minimize

$\Vert A\mathbf{c}-\mathbf{y}{\Vert }^2$. This is a least square problem, and we know that the answer is

$\mathbf{c}=(A^{\mathrm{\top }}A{)}^{-1}{A}^{\mathrm{\top }}\mathbf{y}$.

Sample answer

Let

$$A=\left[\begin{array}{ccc}1& 1& 1\\ 1& 2& 4\\ 1& 3& 9\\ 1& 4& 16\end{array}\right]\text{ and }\mathbf{y}=\left[\begin{array}{c}2\\ 0\\ 0\\ 1\end{array}\right].$$

Then the answer is

$$\begin{array}{rl}\mathbf{c}& =(A^{\mathrm{\top }}A{)}^{-1}{A}^{\mathrm{\top }}\mathbf{y}\\ & =\left[\begin{array}{c}5.25\\ -4.05\\ 0.75\end{array}\right].\end{array}$$

Thus,

$f(x)=5.25-4.05x+0.75x^2$ is the polynomial that best describes the data. One may use desmos to plot the function to see if it is close to the data points.

This note can be found at Course website > Learning resources.