---
title: Ch5-2
tags: Linear algebra
GA: G-77TT93X4N1
---

# Chapter 5 extra note 2

## Selected lecture notes

> metric
> norm
> inner product
> orthogonal/orthonormal
> Pythogorean identity
> Parsevals identity

### Metric space

:::info
**Definition:**
A metric (distance function) on $X$ is a function $d:X\times X\to\mathbb{R}$ such that, $\forall x, y, z\in X$,
1. $d(x, y)\ge 0$;
2. $d(x, y) = d(y, x)$;
3. $d(x, y) \le d(x, z) + d(z, y)$;
4. $d(x, y)=0$ if and only if $x=y$.
:::

#### Example

* Trivial distance:
$$
d(x, y) =
\begin{cases}
1, & \text{if $x\ne y$}, \\
0, & \text{if $x=y$}.
\end{cases}
$$

### Normed vector space

:::info
**Definition:**
A norm on a vector space $V$ is a function $\|\cdot\|:V\to\mathbb{R}$ such that, $\forall {\bf u}, {\bf v}\in V$,
1. $\|\alpha{\bf u}\| = |\alpha|\|{\bf u}\|$;
2. $\|{\bf u}+{\bf v}\| \le \|{\bf u}\|+\|{\bf v}\|$;
3. $\|{\bf u}\|\ge 0$;
4. $\|{\bf u}\|=0$ if and only if ${\bf u}={\bf 0}$.
:::

**Remark**
Given a norm $\|\cdot\|$, we can define
$$
d({\bf u}, {\bf v}) = \|{\bf u} - {\bf v}\|.
$$
Then $d$ is a metric.

:::info
**Definition:**
A vector space with a norm is called a normed space.
:::

:::info
**Definition:**
An inner product space is a normed space.
:::

#### Example 1
Let
$$
{\bf x} = \begin{bmatrix}x_1 \\ x_2\end{bmatrix}\in\mathbb{R}^2.
$$
We define the $p$-norm as
$$
\|{\bf x}\|_p = \left(|x_1|^p + |x_2|^p\right)^{1/p},
$$
so
$$
\|{\bf x}\|_1 = |x_1| + |x_2|, \quad
\|{\bf x}\|_2 = \sqrt{|x_1|^2 + |x_2|^2}.
$$
Besides, we define
$$
\|{\bf x}\|_{\infty} = \max\{|x_1|, |x_2|\}.
$$

#### Example 2
Let
$$
{\bf x} = \begin{bmatrix}x_1 \\ \vdots \\ x_n\end{bmatrix}\in\mathbb{R}^n.
$$
We define the $p$-norm as
$$
\|{\bf x}\|_p = \left(\sum^n_{k=1} |x_k|^p\right)^{1/p},
$$
so
$$
\|{\bf x}\|_1 = \sum^n_{k=1} |x_k|, \quad
\|{\bf x}\|_2 = \sqrt{\sum^n_{k=1}|x_k|^2}.
$$
Also, we define
$$
\|{\bf x}\|_{\infty} = \max_{1\le k\le n}\{|x_k|\}.
$$

#### Example 3
Given $f(x)$, $x\in[a, b]$, we define the $p$-norm as
$$
\|f\|_p = \left(\int^b_a |f(x)|^p\,dx\right)^{1/p},
$$
so
$$
\|f\|_1 = \int^b_a |f(x)|\,dx, \quad
\|f\|_2 = \sqrt{\int^b_a |f(x)|^2\,dx}.
$$
Also, we define
$$
\|{\bf x}\|_{\infty} = \max_{x\in[a, b]}\{|f(x)|\}.
$$

**Remark**
One can see clearly that $2$-norm is induced by usual inner product.

**Remark**
$$
\text{inner product space} \subset \text{normed space} \subset \text{metric space}
$$

---

### Orthogonality

:::info
**Definition:**
${\bf u}$ and ${\bf v}$ are orthogonal if $\langle{\bf u}, {\bf v}\rangle=0$ and we write ${\bf u}\perp{\bf v}$.
:::

**Remark:**
${\bf 0}\perp{\bf u}, \forall{\bf u}\in V$.

**Pythogorean identity:**
If ${\bf u}\perp{\bf v}$, then $\|{\bf u}+{\bf v}\|^2=\|{\bf u}\|^2 + \|{\bf v}\|^2$.
* Proof:
> $$
> \|{\bf u}+{\bf v}\|^2=\langle{\bf u}+{\bf v}, {\bf u}+{\bf v}\rangle = \langle{\bf u}, {\bf u}\rangle+\langle{\bf u}, {\bf v}\rangle+\langle{\bf v}, {\bf u}\rangle+\langle{\bf v}, {\bf v}\rangle.
> $$
> Since ${\bf u}\perp{\bf v}$, $\langle{\bf u}, {\bf v}\rangle=0$ and
> $$
> \langle{\bf v}, {\bf u}\rangle=\overline{\langle{\bf u}, {\bf v}\rangle}=\bar{0} = 0.
> $$
> Therefore,
> $$
> \|{\bf u}+{\bf v}\|^2=\langle{\bf u}, {\bf u}\rangle+\langle{\bf v}, {\bf v}\rangle = \|{\bf u}\|^2 + \|{\bf v}\|^2.
> $$
>

:::info
**Definition:**
$\{{\bf v}_1, \cdots, {\bf v}_n\}$ is orthogonal if ${\bf v}_i\perp{\bf v}_j$ for all $i\ne j$.
:::

**Generalized Pythogorean identity:**
Let $\{{\bf v}_1, \cdots, {\bf v}_n\}$ be an orthogonal set, then
$$
\left\|\sum^n_{k=1} \alpha_k {\bf v}_k\right\|^2 = \sum^n_{k=1}|\alpha_k|^2\|{\bf v}_k\|^2.
$$
* Proof:
> $$
> \begin{align}
> \left\|\sum^n_{k=1} \alpha_k {\bf v}_k\right\|^2 &= \langle\sum^n_{k=1} \alpha_k {\bf v}_k, \sum^n_{k=1} \alpha_k {\bf v}_k\rangle \\
> &= \sum^n_{k=1}\sum^n_{\ell=1}\langle\alpha_k{\bf v}_k, \alpha_{\ell}{\bf v}_{\ell}\rangle\\
> (\text{since $\{v_k\}$ is orthogonal})\,&=\sum^n_{k=1}\langle\alpha_k{\bf v}_k, \alpha_k{\bf v}_k\rangle\\
> &=\sum^n_{k=1}|\alpha_k|^2\|{\bf v}_k\|^2.
> \end{align}
> $$

**Corollary:**
Any orthogonal system of non-zero vectors is linearly independent.
* Proof:
> Let $\{{\bf v}_1, \cdots, {\bf v}_n\}$ be an orthogonal set and ${\bf v}_k\ne {\bf 0}$ $\forall k$.
> Assume $\alpha_1{\bf v}_1 + \cdots \alpha_n{\bf v}_n={\bf 0}$, we have
> $$
> 0 = \|{\bf 0}\|^2 = \|\alpha_1{\bf v}_1 + \cdots \alpha_n{\bf v}_n\|^2 = \sum^n_{k=1}|\alpha_k|^2\|{\bf v}_k\|^2.
> $$
> So we must have $|\alpha_k|\|{\bf v}_k\|=0$, $\forall k$.
> Since ${\bf v}_k\ne{\bf 0}$, it means that $\alpha_k=0$ $\forall k$.

:::info
**Definition:**
$\{{\bf v}_1, \cdots, {\bf v}_n\}$ is orthonormal if it is orthogonal and $\|{\bf v}_k\|=1$ $\forall k$.
:::

**Corollary:**
Any orthonormal system is linearly independent.
* Proof:
> An orthonormal system is orthogonal and does not contains zero vectors, therefore, it is linearly independent.

### Coordinates of vectors

Let $\{{\bf v}_1, \cdots, {\bf v}_n\}$ be an orthogonal set. Assume
$$
{\bf u} = \alpha_1{\bf v}_1 + \cdots + \alpha_n{\bf v}_n,
$$
we have $\langle{\bf u}, {\bf v}_k\rangle = \alpha_k\langle{\bf v}_k, {\bf v}_k\rangle$, that is,
$$
\alpha_k = \frac{\langle{\bf u}, {\bf v}_k\rangle}{\langle{\bf v}_k, {\bf v}_k\rangle}.
$$

**Remark:**
A remarkable fact here is that the coordinates can be calculated directly using the inner product.

Given $\{{\bf v}_1, \cdots, {\bf v}_n\}$ be an orthogonal basis, we have
$$
{\bf u} = \frac{\langle{\bf u}, {\bf v}_1\rangle}{\|{\bf v}_1\|^2}{\bf v}_1 + \cdots + \frac{\langle{\bf u}, {\bf v}_n\rangle}{\|{\bf v}_n\|^2}{\bf v}_n.
$$

#### Example: Fourier series

> I must admit that the following statements are a bit sloppy. We only consider and prove problems with finite dimension, but the following are defined on an infinite dimension.

Recall in Calculus: For $m,n\in\mathbb{N}\cup\{0\}$,
$$
\begin{align}
&\int^{\pi}_{-\pi} \sin(mx)\cos(nx)\,dx = 0,\\
&\int^{\pi}_{-\pi} \sin(mx)\sin(nx)\,dx =
\begin{cases}
0, & \text{if $m\ne n$ or $m=n=0$}, \\
\pi, & \text{if $m=n\ne 0$},
\end{cases}\\
&\int^{\pi}_{-\pi} \cos(mx)\cos(nx)\,dx =
\begin{cases}
0, & \text{if $m\ne n$}, \\
\pi, & \text{if $m=n\ne 0$},\\
2\pi, & \text{if $m=n=0$}.
\end{cases}
\end{align}
$$
That is, under the inner product $\langle f, g\rangle = \int^{\pi}_{-\pi}f(x)g(x)\,dx$, the following set is an orthogonal set:
$$
S = \{1, \sin(x), \cos(x), \sin(2x), \cos(2x), \cdots \}.
$$
> Note that we have removed $\sin(0\cdot x)$ since it's a zero function, and as a result $S$ is linearly independent.

We then have, given $f\in\text{span}\{S\}$,
$$
f(x) = a_0 + \sum^{\infty}_{k=1} a_k\cos(kx) + b_k\sin(kx),
$$
where
$$
\begin{align}
&a_0 = \frac{\langle f, 1\rangle}{\langle 1, 1\rangle} =\frac{1}{2\pi} \int^{\pi}_{-\pi}f(x)\,dx,\\
&a_k = \frac{\langle f, \cos(kx)\rangle}{\langle \cos(kx), \cos(kx)\rangle} =\frac{1}{\pi} \int^{\pi}_{-\pi}f(x)\cos(kx)\,dx,\\
&b_k = \frac{\langle f, \sin(kx)\rangle}{\langle \sin(kx), \sin(kx)\rangle} =\frac{1}{\pi} \int^{\pi}_{-\pi}f(x)\sin(kx)\,dx.
\end{align}
$$

**Parsevals identity:** (extension of generalized Pythagorean identity)
If $f\in\text{span }\{S\}$,
$$
\|f\|^2 = \int^{\pi}_{-\pi} |f(x)|^2\,dx = 2\pi|a_0|^2 + \sum^{\infty}_{k=1} \pi|a_k|^2 + \pi|b_k|^2.
$$

**Corollary:**
If $f\in\text{span }\{S\}$ and $\int^{\pi}_{-\pi} |f(x)|^2\,dx<\infty$, then $\sum^{\infty}_{k=1} |a_k|^2<\infty$ and $\sum^{\infty}_{k=1} |b_k|^2<\infty$, which implies
$$
\lim_{k\to\infty} a_k=0, \quad \lim_{k\to\infty} b_k=0.
$$