Chapter 5 extra note 2

Selected lecture notes

metric
norm
inner product
orthogonal/orthonormal
Pythogorean identity
Parsevals identity

Metric space

Definition:
A metric (distance function) on

$X$ is a function

$d:X\times X\to \mathbb{R}$ such that,

$\mathrm{\forall }x,y,z\in X$,

1. $d(x,y)\ge 0$;
2. $d(x,y)=d(y,x)$;
3. $d(x,y)\le d(x,z)+d(z,y)$;
4. $d(x,y)=0$ if and only if
   $x=y$.

Example

• Trivial distance:
  
  $$d(x,y)=\{\begin{array}{ll}1,& \text{if }x\ne y,\\ 0,& \text{if }x=y.\end{array}$$

Normed vector space

Definition:
A norm on a vector space

$V$ is a function

$\Vert \cdot \Vert :V\to \mathbb{R}$ such that,

$\mathrm{\forall }\mathbf{u},\mathbf{v}\in V$,

1. $\Vert \alpha \mathbf{u}\Vert =|\alpha |\Vert \mathbf{u}\Vert$;
2. $\Vert \mathbf{u}+\mathbf{v}\Vert \le \Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert$;
3. $\Vert \mathbf{u}\Vert \ge 0$;
4. $\Vert \mathbf{u}\Vert =0$ if and only if
   $\mathbf{u}=\mathbf{0}$.

Remark
Given a norm

$\Vert \cdot \Vert$, we can define

$$d(\mathbf{u},\mathbf{v})=\Vert \mathbf{u}-\mathbf{v}\Vert .$$
Then

$d$ is a metric.

Definition:
A vector space with a norm is called a normed space.

Definition:
An inner product space is a normed space.

Example 1

Let

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]\in {\mathbb{R}}^2.$$
We define the

$p$-norm as

$$\Vert \mathbf{x}{\Vert }_p={(|x_1{|}^p+|x_2{|}^p)}^{1/p},$$
so

$$\Vert \mathbf{x}{\Vert }_1=|x_1|+|x_2|,\Vert \mathbf{x}{\Vert }_2=\sqrt{|x_1{|}^2+|x_2{|}^2}.$$
Besides, we define

$$\Vert \mathbf{x}{\Vert }_{\mathrm{\infty }}=max\{|x_1|,|x_2|\}.$$

Example 2

Let

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_n\end{array}\right]\in {\mathbb{R}}^n.$$
We define the

$p$-norm as

$$\Vert \mathbf{x}{\Vert }_p={\left(\sum _{k=1}^n|x_k{|}^p\right)}^{1/p},$$
so

$$\Vert \mathbf{x}{\Vert }_1=\sum _{k=1}^n|x_k|,\Vert \mathbf{x}{\Vert }_2=\sqrt{\sum _{k=1}^n|x_k{|}^2}.$$
Also, we define

$$\Vert \mathbf{x}{\Vert }_{\mathrm{\infty }}=\underset{1\le k\le n}{max}\{|x_k|\}.$$

Example 3

Given

$f(x)$,

$x\in [a,b]$, we define the

$p$-norm as

$$\Vert f{\Vert }_p={({\int }_a^b|f(x){|}^{p}dx)}^{1/p},$$
so

$$\Vert f{\Vert }_1={\int }_a^b|f(x)|dx,\Vert f{\Vert }_2=\sqrt{{\int }_a^b|f(x){|}^{2}dx}.$$
Also, we define

$$\Vert \mathbf{x}{\Vert }_{\mathrm{\infty }}=\underset{x\in [a,b]}{max}\{|f(x)|\}.$$

Remark
One can see clearly that

$2$-norm is induced by usual inner product.

Remark

$$\text{inner product space}\subset \text{normed space}\subset \text{metric space}$$

---

Orthogonality

Definition:

$\mathbf{u}$ and

$\mathbf{v}$ are orthogonal if

$⟨\mathbf{u},\mathbf{v}⟩=0$ and we write

$\mathbf{u}\perp \mathbf{v}$.

Remark:

$\mathbf{0}\perp \mathbf{u},\mathrm{\forall }\mathbf{u}\in V$.

Pythogorean identity:
If

$\mathbf{u}\perp \mathbf{v}$, then

$\Vert \mathbf{u}+\mathbf{v}{\Vert }^2=\Vert \mathbf{u}{\Vert }^2+\Vert \mathbf{v}{\Vert }^2$.

• Proof:

  $$\Vert \mathbf{u}+\mathbf{v}{\Vert }^2=⟨\mathbf{u}+\mathbf{v},\mathbf{u}+\mathbf{v}⟩=⟨\mathbf{u},\mathbf{u}⟩+⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{v},\mathbf{u}⟩+⟨\mathbf{v},\mathbf{v}⟩.$$
  Since

  $\mathbf{u}\perp \mathbf{v}$,

  $⟨\mathbf{u},\mathbf{v}⟩=0$ and
  

  $$⟨\mathbf{v},\mathbf{u}⟩=\overline{⟨\mathbf{u},\mathbf{v}⟩}=\overline{0}=0.$$
  Therefore,
  

  $$\Vert \mathbf{u}+\mathbf{v}{\Vert }^2=⟨\mathbf{u},\mathbf{u}⟩+⟨\mathbf{v},\mathbf{v}⟩=\Vert \mathbf{u}{\Vert }^2+\Vert \mathbf{v}{\Vert }^2.$$

Definition:

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ is orthogonal if

${\mathbf{v}}_i\perp {\mathbf{v}}_j$ for all

$i\ne j$.

Generalized Pythogorean identity:
Let

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be an orthogonal set, then

$${\Vert \sum _{k=1}^n{\alpha }_k{\mathbf{v}}_k\Vert }^2=\sum _{k=1}^n|{\alpha }_k{|}^2\Vert {\mathbf{v}}_k{\Vert }^2.$$

• Proof:

  $$\begin{array}{rl}{\Vert \sum _{k=1}^n{\alpha }_k{\mathbf{v}}_k\Vert }^2& =⟨\sum _{k=1}^n{\alpha }_k{\mathbf{v}}_k,\sum _{k=1}^n{\alpha }_k{\mathbf{v}}_k⟩\\ & =\sum _{k=1}^n\sum _{\ell =1}^n⟨{\alpha }_k{\mathbf{v}}_k,{\alpha }_{\ell }{\mathbf{v}}_{\ell }⟩\\ (\text{since }\{v_k\}\text{ is orthogonal})& =\sum _{k=1}^n⟨{\alpha }_k{\mathbf{v}}_k,{\alpha }_k{\mathbf{v}}_k⟩\\ & =\sum _{k=1}^n|{\alpha }_k{|}^2\Vert {\mathbf{v}}_k{\Vert }^2.\end{array}$$

Corollary:
Any orthogonal system of non-zero vectors is linearly independent.

• Proof:

  Let

  $\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be an orthogonal set and

  ${\mathbf{v}}_k\ne \mathbf{0}$

  $\mathrm{\forall }k$.
  Assume

  ${\alpha }_1{\mathbf{v}}_1+\cdots {\alpha }_n{\mathbf{v}}_n=\mathbf{0}$, we have
  

  $$0=\Vert \mathbf{0}{\Vert }^2=\Vert {\alpha }_1{\mathbf{v}}_1+\cdots {\alpha }_n{\mathbf{v}}_n{\Vert }^2=\sum _{k=1}^n|{\alpha }_k{|}^2\Vert {\mathbf{v}}_k{\Vert }^2.$$
  So we must have

  $|{\alpha }_k|\Vert {\mathbf{v}}_k\Vert =0$,

  $\mathrm{\forall }k$.
  Since

  ${\mathbf{v}}_k\ne \mathbf{0}$, it means that

  ${\alpha }_k=0$

  $\mathrm{\forall }k$.

Definition:

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ is orthonormal if it is orthogonal and

$\Vert {\mathbf{v}}_k\Vert =1$

$\mathrm{\forall }k$.

Corollary:
Any orthonormal system is linearly independent.

• Proof:

  An orthonormal system is orthogonal and does not contains zero vectors, therefore, it is linearly independent.

Coordinates of vectors

Let

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be an orthogonal set. Assume

$$\mathbf{u}={\alpha }_1{\mathbf{v}}_1+\cdots +{\alpha }_n{\mathbf{v}}_n,$$
we have

$⟨\mathbf{u},{\mathbf{v}}_k⟩={\alpha }_k⟨{\mathbf{v}}_k,{\mathbf{v}}_k⟩$, that is,

$${\alpha }_k=\frac{⟨\mathbf{u},{\mathbf{v}}_k⟩}{⟨{\mathbf{v}}_k,{\mathbf{v}}_k⟩}.$$

Remark:
A remarkable fact here is that the coordinates can be calculated directly using the inner product.

Given

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be an orthogonal basis, we have

$$\mathbf{u}=\frac{⟨\mathbf{u},{\mathbf{v}}_1⟩}{\Vert {\mathbf{v}}_1{\Vert }^2}{\mathbf{v}}_1+\cdots +\frac{⟨\mathbf{u},{\mathbf{v}}_n⟩}{\Vert {\mathbf{v}}_n{\Vert }^2}{\mathbf{v}}_n.$$

Example: Fourier series

I must admit that the following statements are a bit sloppy. We only consider and prove problems with finite dimension, but the following are defined on an infinite dimension.

Recall in Calculus: For

$m,n\in \mathbb{N}\cup \{0\}$,

$$\begin{array}{rl}& {\int }_{-\pi }^{\pi }\sin (mx)\cos (nx)dx=0,\\ & {\int }_{-\pi }^{\pi }\sin (mx)\sin (nx)dx=\{\begin{array}{ll}0,& \text{if }m\ne n\text{ or }m=n=0,\\ \pi ,& \text{if }m=n\ne 0,\end{array}\\ & {\int }_{-\pi }^{\pi }\cos (mx)\cos (nx)dx=\{\begin{array}{ll}0,& \text{if }m\ne n,\\ \pi ,& \text{if }m=n\ne 0,\\ 2\pi ,& \text{if }m=n=0.\end{array}\end{array}$$
That is, under the inner product

$⟨f,g⟩={\int }_{-\pi }^{\pi }f(x)g(x)dx$, the following set is an orthogonal set:

$$S=\{1,\sin (x),\cos (x),\sin (2x),\cos (2x),\cdots \}.$$

Note that we have removed

$\sin (0\cdot x)$ since it's a zero function, and as a result

$S$ is linearly independent.

We then have, given

$f\in \text{span}\{S\}$,

$$f(x)=a_0+\sum _{k=1}^{\mathrm{\infty }}{a}_k\cos (kx)+b_k\sin (kx),$$
where

$$\begin{array}{rl}& a_0=\frac{⟨f,1⟩}{⟨1,1⟩}=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }f(x)dx,\\ & a_k=\frac{⟨f,\cos (kx)⟩}{⟨\cos (kx),\cos (kx)⟩}=\frac{1}{\pi }{\int }_{-\pi }^{\pi }f(x)\cos (kx)dx,\\ & b_k=\frac{⟨f,\sin (kx)⟩}{⟨\sin (kx),\sin (kx)⟩}=\frac{1}{\pi }{\int }_{-\pi }^{\pi }f(x)\sin (kx)dx.\end{array}$$

Parsevals identity: (extension of generalized Pythagorean identity)
If

$f\in \text{span }\{S\}$,

$$\Vert f{\Vert }^2={\int }_{-\pi }^{\pi }|f(x){|}^{2}dx=2\pi |a_0{|}^2+\sum _{k=1}^{\mathrm{\infty }}\pi |a_k{|}^2+\pi |b_k{|}^2.$$

Corollary:
If

$f\in \text{span }\{S\}$ and

${\int }_{-\pi }^{\pi }|f(x){|}^{2}dx<\mathrm{\infty }$, then

$\sum _{k=1}^{\mathrm{\infty }}|a_k{|}^2<\mathrm{\infty }$ and

$\sum _{k=1}^{\mathrm{\infty }}|b_k{|}^2<\mathrm{\infty }$, which implies

$$\underset{k\to \mathrm{\infty }}{lim}{a}_k=0,\underset{k\to \mathrm{\infty }}{lim}{b}_k=0.$$