---
title: Ch5-3
tags: Linear algebra
GA: G-77TT93X4N1
---

# Chapter 5 extra note 3

## Selected lecture notes

> orthogonal projection
> Gram-Schmidt method

### Orthogonal projection

:::info
**Definition:**
Let $E\subset V$ be a vector subspace and ${\bf u}\in V$ be a vector, then ${\bf u}$ is orthogonal to $E$ if $\langle{\bf u}, {\bf v}\rangle=0$ for all ${\bf v}\in E$, and we write denoted by ${\bf u}\perp E$.
:::

**Lemma:**
Let $E = \text{span}\{{\bf v}_1, \cdots, {\bf v}_n\}$, then ${\bf u}\perp E$ if and only if ${\bf u}\perp {\bf v}_k$ for all $k$.

:::info
**Definition:**
Let $E,F\subset V$ be vector subspaces, then $E\perp F$ if $\langle{\bf u}, {\bf v}\rangle=0$ for all ${\bf u}\in E$ and ${\bf v}\in F$.
:::

:::info
**Definition:**
The orthogonal projection of a vector ${\bf u}$ on a vector subspace $E$ is a vector ${\bf w}$ satisfies
1. ${\bf w}\in E$;
2. $({\bf u}-{\bf w})\perp E$.
:::

**Theorem:**
Let ${\bf w}$ be the orthogonal projection of ${\bf u}$, then
$$
\|{\bf u}-{\bf w}\|\le \|{\bf u}-{\bf v}\|, \quad \forall {\bf v}\in E,
$$
and the equality holds if ${\bf v}={\bf w}$.
* Proof:
> Given ${\bf v}\in E$,
> $$
> {\bf u}-{\bf v} = ({\bf u}-{\bf w}) + ({\bf w}-{\bf v}).
> $$
> Since ${\bf w, v}\in E$, we have $({\bf w}-{\bf v})\in E$.
> Since ${\bf w}$ is the orthogonal projection of ${\bf u}$, we have $({\bf u}-{\bf w})\perp E$, and hence $({\bf u}-{\bf w})\perp ({\bf w}-{\bf v})$.
>
> By Pythagorean theorem, we obtain
> $$
> \|{\bf u}-{\bf v}\|^2 = \|{\bf u}-{\bf w}\|^2 + \|{\bf w}-{\bf v}\|^2.
> $$
> Therefore
> $$
> \|{\bf u}-{\bf w}\|^2 \le \|{\bf u}-{\bf v}\|^2,
> $$
> and the equality holds if $\|{\bf w}-{\bf v}\|^2=0$, which means ${\bf v}={\bf w}$.

**Remarks:**
The Theorem ensures the ***uniqueness*** of the projection, and therefore we can denote the orthogonal projection of a vector ${\bf u}$ on a vector subspace $E$ by $P_E{\bf u}$.

* Proof of the uniqueness:
>
> Suppose there exists ${\bf w}_2$ that is also an orthogonal projection of ${\bf u}$ on $E$. That is, ${\bf w}_2\in E$ and $({\bf u}-{\bf w}_2)\perp E$.
>
> Then, since ${\bf w}_2\in E$, according to the above theorem we have
> $$
> \|{\bf u}-{\bf w}\|\le \|{\bf u}-{\bf w}_2\|.
> $$
> Similarly, ${\bf w}_2\in E$ and $({\bf u}-{\bf w}_2)\perp E$ implies
> $$
> \|{\bf u}-{\bf w}_2\|\le \|{\bf u}-{\bf v}\|, \quad \forall {\bf v}\in E,
> $$
> so we must have
> $$
> \|{\bf u}-{\bf w}_2\|\le \|{\bf u}-{\bf w}\|.
> $$
> Therefore, $\|{\bf u}-{\bf w}_2\|=\|{\bf u}-{\bf w}\|$, and, based on the Theorem again, ${\bf w}={\bf w}_2$.

**Proposition:**
Let $\{{\bf v}_1, \cdots, {\bf v}_n\}$ be an orthogonal basis of a vector subspace $E$, then, given ${\bf u}\in V$,
$$
P_E{\bf u} = \sum^n_{k=1} \frac{\langle{\bf u}, {\bf v}_k\rangle}{\langle{\bf v}_k, {\bf v}_k\rangle}{\bf v}_k.
$$
* Proof:
> 1. $P_E{\bf u}$ is a linear combination of ${\bf v}_k$'s, so $P_E{\bf u}\in E$.
> 2.
> $$
> \langle{\bf u}-{\bf w}, {\bf v}_j\rangle = \langle{\bf u}, {\bf v}_j\rangle - \langle{\bf w}, {\bf v}_j\rangle.
> $$
> We also have, since ${\bf v}_k$'s are orthogonal,
> $$
> \langle{\bf w}, {\bf v}_j\rangle = \langle P_E{\bf u}, {\bf v}_j\rangle = \langle\frac{\langle{\bf u}, {\bf v}_j\rangle}{\langle{\bf v}_j, {\bf v}_j\rangle}{\bf v}_j, {\bf v}_j\rangle = \left(\frac{\langle{\bf u}, {\bf v}_j\rangle}{\langle{\bf v}_j, {\bf v}_j\rangle}\right)\langle{\bf v}_j, {\bf v}_j\rangle = \langle{\bf u}, {\bf v}_j\rangle.
> $$
> Therefore, $\langle{\bf u}-{\bf w}, {\bf v}_j\rangle=0$ for all $j$, and we have $({\bf u}-{\bf w})\perp E$.

**Remarks:**
The Proposition ensures the ***existence*** of the projection.

### Gram-Schmidt method

Given $S=\{{\bf u}_1, \cdots, {\bf u}_n\}$ be a linearly independent set. We can use Gram-Schmidt process to generate an orthogonal set $\beta=\{{\bf v}_1, \cdots, {\bf v}_n\}$ such that $\text{span}\{\beta\} = \text{span}\{S\}$.

**Gram-Schmidt process (the basic idea):**
1. ${\bf v}_1 = {\bf u}_1$, $\quad E_1 = \text{span}\{{\bf v}_1\}$.
2. ${\bf v}_2 = {\bf u}_2 - P_{E_1}{\bf u}_2$, $\quad E_2 = \text{span}\{{\bf v}_1, {\bf v}_2\}$.
3. ${\bf v}_3 = {\bf u}_3 - P_{E_2}{\bf u}_3$, $\quad E_3 = \text{span}\{{\bf v}_1, {\bf v}_2, {\bf v}_3\}$.
4. $\cdots$

**Gram-Schmidt process (algorithm):**
$$
\begin{align}
{\bf v}_1 &= {\bf u}_1;\\
{\bf v}_2 &= {\bf u}_2 - \frac{\langle{\bf u}_2, {\bf v}_1\rangle}{\langle{\bf v}_1, {\bf v}_1\rangle}{\bf v}_1;\\
{\bf v}_3 &= {\bf u}_3 - \frac{\langle{\bf u}_3, {\bf v}_1\rangle}{\langle{\bf v}_1, {\bf v}_1\rangle}{\bf v}_1- \frac{\langle{\bf u}_3, {\bf v}_2\rangle}{\langle{\bf v}_2, {\bf v}_2\rangle}{\bf v}_2;\\
\vdots
\end{align}
$$