Chapter 5 extra note 3

Selected lecture notes

orthogonal projection
Gram-Schmidt method

Orthogonal projection

Definition:
Let

$E\subset V$ be a vector subspace and

$\mathbf{u}\in V$ be a vector, then

$\mathbf{u}$ is orthogonal to

$E$ if

$⟨\mathbf{u},\mathbf{v}⟩=0$ for all

$\mathbf{v}\in E$, and we write denoted by

$\mathbf{u}\perp E$.

Lemma:
Let

$E=\text{span}\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$, then

$\mathbf{u}\perp E$ if and only if

$\mathbf{u}\perp {\mathbf{v}}_k$ for all

$k$.

Definition:
Let

$E,F\subset V$ be vector subspaces, then

$E\perp F$ if

$⟨\mathbf{u},\mathbf{v}⟩=0$ for all

$\mathbf{u}\in E$ and

$\mathbf{v}\in F$.

Definition:
The orthogonal projection of a vector

$\mathbf{u}$ on a vector subspace

$E$ is a vector

$\mathbf{w}$ satisfies

1. $\mathbf{w}\in E$;
2. $(\mathbf{u}-\mathbf{w})\perp E$.

Theorem:
Let

$\mathbf{w}$ be the orthogonal projection of

$\mathbf{u}$, then

$$\Vert \mathbf{u}-\mathbf{w}\Vert \le \Vert \mathbf{u}-\mathbf{v}\Vert ,\mathrm{\forall }\mathbf{v}\in E,$$
and the equality holds if

$\mathbf{v}=\mathbf{w}$.

• Proof:

  Given

  $\mathbf{v}\in E$,
  

  $$\mathbf{u}-\mathbf{v}=(\mathbf{u}-\mathbf{w})+(\mathbf{w}-\mathbf{v}).$$
  Since

  $\mathbf{w}\mathbf{,}\mathbf{v}\in E$, we have

  $(\mathbf{w}-\mathbf{v})\in E$.
  Since

  $\mathbf{w}$ is the orthogonal projection of

  $\mathbf{u}$, we have

  $(\mathbf{u}-\mathbf{w})\perp E$, and hence

  $(\mathbf{u}-\mathbf{w})\perp (\mathbf{w}-\mathbf{v})$.

  By Pythagorean theorem, we obtain
  

  $$\Vert \mathbf{u}-\mathbf{v}{\Vert }^2=\Vert \mathbf{u}-\mathbf{w}{\Vert }^2+\Vert \mathbf{w}-\mathbf{v}{\Vert }^2.$$
  Therefore
  

  $$\Vert \mathbf{u}-\mathbf{w}{\Vert }^2\le \Vert \mathbf{u}-\mathbf{v}{\Vert }^2,$$
  and the equality holds if

  $\Vert \mathbf{w}-\mathbf{v}{\Vert }^2=0$, which means

  $\mathbf{v}=\mathbf{w}$.

Remarks:
The Theorem ensures the uniqueness of the projection, and therefore we can denote the orthogonal projection of a vector

$\mathbf{u}$ on a vector subspace

$E$ by

$P_E\mathbf{u}$.

• Proof of the uniqueness:

  Suppose there exists

  ${\mathbf{w}}_2$ that is also an orthogonal projection of

  $\mathbf{u}$ on

  $E$. That is,

  ${\mathbf{w}}_2\in E$ and

  $(\mathbf{u}-{\mathbf{w}}_2)\perp E$.

  Then, since

  ${\mathbf{w}}_2\in E$, according to the above theorem we have
  

  $$\Vert \mathbf{u}-\mathbf{w}\Vert \le \Vert \mathbf{u}-{\mathbf{w}}_2\Vert .$$
  Similarly,

  ${\mathbf{w}}_2\in E$ and

  $(\mathbf{u}-{\mathbf{w}}_2)\perp E$ implies
  

  $$\Vert \mathbf{u}-{\mathbf{w}}_2\Vert \le \Vert \mathbf{u}-\mathbf{v}\Vert ,\mathrm{\forall }\mathbf{v}\in E,$$
  so we must have
  

  $$\Vert \mathbf{u}-{\mathbf{w}}_2\Vert \le \Vert \mathbf{u}-\mathbf{w}\Vert .$$
  Therefore,

  $\Vert \mathbf{u}-{\mathbf{w}}_2\Vert =\Vert \mathbf{u}-\mathbf{w}\Vert$, and, based on the Theorem again,

  $\mathbf{w}={\mathbf{w}}_2$.

Proposition:
Let

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be an orthogonal basis of a vector subspace

$E$, then, given

$\mathbf{u}\in V$,

$$P_E\mathbf{u}=\sum _{k=1}^n\frac{⟨\mathbf{u},{\mathbf{v}}_k⟩}{⟨{\mathbf{v}}_k,{\mathbf{v}}_k⟩}{\mathbf{v}}_k.$$

• Proof:

  1. $P_E\mathbf{u}$ is a linear combination of
     ${\mathbf{v}}_k$'s, so
     $P_E\mathbf{u}\in E$.

  $$⟨\mathbf{u}-\mathbf{w},{\mathbf{v}}_j⟩=⟨\mathbf{u},{\mathbf{v}}_j⟩-⟨\mathbf{w},{\mathbf{v}}_j⟩.$$
  We also have, since

  ${\mathbf{v}}_k$'s are orthogonal,
  

  $$⟨\mathbf{w},{\mathbf{v}}_j⟩=⟨P_E\mathbf{u},{\mathbf{v}}_j⟩=⟨\frac{⟨\mathbf{u},{\mathbf{v}}_j⟩}{⟨{\mathbf{v}}_j,{\mathbf{v}}_j⟩}{\mathbf{v}}_j,{\mathbf{v}}_j⟩=\left(\frac{⟨\mathbf{u},{\mathbf{v}}_j⟩}{⟨{\mathbf{v}}_j,{\mathbf{v}}_j⟩}\right)⟨{\mathbf{v}}_j,{\mathbf{v}}_j⟩=⟨\mathbf{u},{\mathbf{v}}_j⟩.$$
  Therefore,

  $⟨\mathbf{u}-\mathbf{w},{\mathbf{v}}_j⟩=0$ for all

  $j$, and we have

  $(\mathbf{u}-\mathbf{w})\perp E$.

Remarks:
The Proposition ensures the existence of the projection.

Gram-Schmidt method

Given

$S=\{{\mathbf{u}}_1,\cdots ,{\mathbf{u}}_n\}$ be a linearly independent set. We can use Gram-Schmidt process to generate an orthogonal set

$\beta =\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ such that

$\text{span}\{\beta \}=\text{span}\{S\}$.

Gram-Schmidt process (the basic idea):

1. ${\mathbf{v}}_1={\mathbf{u}}_1$,
   $E_1=\text{span}\{{\mathbf{v}}_1\}$.
2. ${\mathbf{v}}_2={\mathbf{u}}_2-P_{E_1}{\mathbf{u}}_2$,
   $E_2=\text{span}\{{\mathbf{v}}_1,{\mathbf{v}}_2\}$.
3. ${\mathbf{v}}_3={\mathbf{u}}_3-P_{E_2}{\mathbf{u}}_3$,
   $E_3=\text{span}\{{\mathbf{v}}_1,{\mathbf{v}}_2,{\mathbf{v}}_3\}$.
4. $\cdots$

Gram-Schmidt process (algorithm):

$$\begin{array}{rl}{\mathbf{v}}_1& ={\mathbf{u}}_1;\\ {\mathbf{v}}_2& ={\mathbf{u}}_2-\frac{⟨{\mathbf{u}}_2,{\mathbf{v}}_1⟩}{⟨{\mathbf{v}}_1,{\mathbf{v}}_1⟩}{\mathbf{v}}_1;\\ {\mathbf{v}}_3& ={\mathbf{u}}_3-\frac{⟨{\mathbf{u}}_3,{\mathbf{v}}_1⟩}{⟨{\mathbf{v}}_1,{\mathbf{v}}_1⟩}{\mathbf{v}}_1-\frac{⟨{\mathbf{u}}_3,{\mathbf{v}}_2⟩}{⟨{\mathbf{v}}_2,{\mathbf{v}}_2⟩}{\mathbf{v}}_2;\\ ⋮\end{array}$$