---
title: Ch5-1
tags: Linear algebra
GA: G-77TT93X4N1
---

# Chapter 5 extra note 1

## Selected lecture notes

> inner product
> norm
> Cauchy-Schwartz inequality
> Triangular inequality

### Inner product space

:::info
**Definition:**
An inner product on a vector space $V$ is a function that assign to each pair of vectors ${\bf u, v}\in V$ a scalar, denoted by $\langle{\bf u}, {\bf v}\rangle$, such that the followings hold: ($\forall {\bf u}, {\bf v}, {\bf w}\in V$, $\alpha, \beta\in F$)
1. $\langle{\bf u}, {\bf v}\rangle = \overline{\langle{\bf v}, {\bf u}\rangle}$, where the overline denotes complex conjugate.
2. $\langle\alpha{\bf u}+\beta{\bf v}, {\bf w}\rangle = \alpha\langle{\bf u}, {\bf w}\rangle+\beta\langle{\bf v}, {\bf w}\rangle$.
3. $\langle{\bf u}, {\bf u}\rangle\ge 0$.
4. $\langle{\bf u}, {\bf u}\rangle= 0$ if and only if ${\bf u}={\bf 0}$.
:::

:::info
**Definition:**
A vector space together with an inner product is called an inner product space.
:::

**Proposition**
(2'). $\langle{\bf u}, \alpha{\bf v}+\beta{\bf w}\rangle = \bar{\alpha}\langle{\bf u}, {\bf v}\rangle+\bar{\beta}\langle{\bf u}, {\bf w}\rangle$.
* Proof:
> $$
> \begin{align}
> \langle{\bf u}, \alpha{\bf v}+\beta{\bf w}\rangle &= \overline{\langle\alpha{\bf v}+\beta{\bf w}, {\bf u}\rangle}\\
> &=\overline{\alpha\langle{\bf v}, {\bf u}\rangle+\beta\langle{\bf w}, {\bf u}\rangle}\\
> &=\overline{\alpha\overline{\langle{\bf u}, {\bf v}\rangle}+\beta\overline{\langle{\bf u}, {\bf w}\rangle}}\\
> &=\bar{\alpha}\langle{\bf u}, {\bf v}\rangle+\bar{\beta}\langle{\bf u}, {\bf w}\rangle.
> \end{align}
> $$
>

:::info
**Definition:**
Given an inner product space, we define the **norm** by
$$
\|{\bf u}\| = \sqrt{\langle{\bf u}, {\bf u}\rangle}.
$$
:::

**Theorem: (Cauchy-Schwartz inequality)**
$$
\left|\langle{\bf u}, {\bf v}\rangle\right|\le \|{\bf u}\|\|{\bf v}\|.
$$
* Proof:
> If ${\bf v}={\bf 0}$, then $\langle{\bf u}, {\bf v}\rangle=0$ and $\|{\bf v}\|=0$. Done.
>
> Assume ${\bf v}\ne{\bf 0}$, given $t\in \mathbb{C}$,
> $$
> \begin{align}
> 0\le \|{\bf u} - t{\bf v}\|^2 &= \langle{\bf u} - t{\bf v}, {\bf u} - t{\bf v}\rangle\\
> &=\langle{\bf u}, {\bf u}\rangle - \langle{\bf u}, t{\bf v}\rangle- \langle t{\bf v}, {\bf u}\rangle + |t|^2\langle{\bf v}, {\bf v}\rangle\\
> &=\|{\bf u}\|^2 -\bar{t}\langle {\bf u}, {\bf v}\rangle - t\overline{\langle {\bf u}, {\bf v}\rangle} + |t|^2\|{\bf v}\|^2.
> \end{align}
> $$
> Let $t=\frac{\langle{\bf u}, {\bf v}\rangle}{\|{\bf v}\|^2}$, then
> $$
> \begin{align}
> 0\le \|{\bf u} - t{\bf v}\|^2 &= \|{\bf u}\|^2 - 2\frac{|\langle{\bf u}, {\bf v}\rangle|^2}{\|{\bf v}\|^2} + \frac{|\langle{\bf u}, {\bf v}\rangle|^2}{\|{\bf v}\|^2}\\
> &= \|{\bf u}\|^2 - \frac{|\langle{\bf u}, {\bf v}\rangle|^2}{\|{\bf v}\|^2}.
> \end{align}
> $$
> Therefore,
> $$
> \left|\langle{\bf u}, {\bf v}\rangle\right|\le \|{\bf u}\|\|{\bf v}\|.
> $$

**Remark:**
The equality of Cauchy-Schwartz inequality holds when $\|{\bf u} - t{\bf v}\|=0$, i.e., ${\bf u} - t{\bf v}={\bf 0}$ and ${\bf u} = t{\bf v}$. That is, when ${\bf u}$ and ${\bf v}$ are "parallel" to each other.

**Lemma:**
Let $V$ be an inner product space, then ${\bf u}={\bf 0}$ iff $\langle{\bf u}, {\bf v}\rangle =0$ $\forall {\bf v}\in V$.
* Proof:
> ($\Rightarrow$)
> Let ${\bf u}={\bf 0}$, given ${\bf v}\in V$,
> $$
> \langle{\bf u}, {\bf v}\rangle = \langle{\bf 0}, {\bf v}\rangle = \langle 2\cdot{\bf 0}, {\bf v}\rangle = 2\langle{\bf 0}, {\bf v}\rangle.
> $$
> So
> $$
> 0 = 2\langle{\bf 0}, {\bf v}\rangle - \langle{\bf 0}, {\bf v}\rangle = 2\langle{\bf 0}, {\bf v}\rangle - \langle 1\cdot{\bf 0}, {\bf v}\rangle = 2\langle{\bf 0}, {\bf v}\rangle - 1\langle{\bf 0}, {\bf v}\rangle = 1\langle{\bf 0}, {\bf v}\rangle = \langle{\bf 0}, {\bf v}\rangle.
> $$
>
> ($\Leftarrow$)
> Assume $\langle{\bf u}, {\bf v}\rangle =0$ $\forall {\bf v}\in V$. Choose ${\bf v}={\bf u}$ we have
> $$
> \langle{\bf u}, {\bf u}\rangle =0.
> $$
> Based on condition 4 of the inner product space, we have ${\bf u}={\bf 0}$.

**Corollary:**
Let ${\bf u}, {\bf v}\in V$ where $V$ is an inner product space, then ${\bf u}={\bf v}$ iff $\langle{\bf u}, {\bf z}\rangle =\langle{\bf v}, {\bf z}\rangle$ $\forall {\bf z}\in V$.

**Lemma: (Triangular inequality)**
$\forall {\bf u, v}\in V$ where $V$ is an inner product space, we have
$$
\| {\bf u} +{\bf v}\| \le \|{\bf u}\| + \|{\bf v}\|.
$$
* Proof:
> $$
> \begin{align}
> \|{\bf u} +{\bf v}\|^2 &= \langle{\bf u}+{\bf v}, {\bf u}+{\bf v}\rangle\\
> &= \|{\bf u}\|^2 +\langle{\bf u}, {\bf v}\rangle+\langle{\bf v}, {\bf u}\rangle + \|{\bf v}\|^2\\
> &=\|{\bf u}\|^2 +2\text{Re}\langle{\bf u}, {\bf v}\rangle+ \|{\bf v}\|^2\\
> &\le \|{\bf u}\|^2 +2|\langle{\bf u}, {\bf v}\rangle|+ \|{\bf v}\|^2\\
> &\le \|{\bf u}\|^2 +2\|{\bf u}\|\|{\bf v}\|+ \|{\bf v}\|^2\\
> &= \left(\|{\bf u}\| + \|{\bf v}\|\right)^2.
> \end{align}
> $$
>

## Examples

### Example 1

In $\mathbb{R}^2$, show that the following defines an inner product
$$
\langle(x_1, x_2), (y_1, y_2)\rangle = x_1y_1+2x_2y_2.
$$
* Proof:
> (1) $\langle(x_1, x_2), (y_1, y_2)\rangle = x_1y_1+2x_2y_2=\langle(y_1, y_2), (x_1, x_2)\rangle$.
> (2)
> $$
> \begin{align}
> \langle\alpha (x_1, x_2) + \beta (y_1, y_2), (z_1, z_2)\rangle &= (\alpha x_1+\beta y_1)z_1 + 2(\alpha x_2+\beta y_2)z_2\\
> &= \alpha(x_1z_1 + 2x_2z_2) + \beta(y_1z_1 + 2y_2z_2)\\
> &=\alpha\langle(x_1, x_2), (z_1, z_2)\rangle + \beta \langle(y_1, y_2), (z_1, z_2)\rangle.
> \end{align}
> $$
> (3) $\langle(x_1, x_2), (x_1, x_2)\rangle=x^2_1 + 2x_2^2\ge 0$.
> (4) If $\langle(x_1, x_2), (x_1, x_2)\rangle=x^2_1 + 2x_2^2= 0$, then $x_1=x_2=0$, i.e., $(x_1, x_2)=(0, 0)$.

With this inner product, the norm is defined as
$$
\|(x_1, x_2)\| = \sqrt{x^2_1 + 2x^2_2}.
$$

We can then apply Cauchy Schwartz inequality to have
$$
(x_1y_1 + 2x_2y_2)^2 = |\langle(x_1, x_2), (y_1, y_2)\rangle|^2 \le \|(x_1, x_2)\|^2\|(y_1, y_2)\|^2=(x^2_1 + 2x^2_2)(y^2_1 + 2y^2_2).
$$

### Example 2
In $\mathbb{P}_n([-1, 1])$, we can define the inner product as
$$
\langle f(x), g(x)\rangle = \int^1_{-1}f(x)g(x)\,dx.
$$
If we allow polynomials with complex coefficients, the inner product should be defined as
$$
\langle f(x), g(x)\rangle = \int^1_{-1}f(x)\overline{g(x)}\,dx.
$$

The norm is then defined by
$$
\|f(x)\| = \sqrt{\int^1_{-1}|f(x)|^2\,dx},
$$
and the Cauchy Schwartz inequality reads
$$
\left(\int^1_{-1}f(x)\overline{g(x)}\,dx\right)^2 = |\langle f, g\rangle|^2 \le \|f\|^2\|g\|^2=\left(\int^1_{-1}|f|^2\,dx\right)\left(\int^1_{-1}|g|^2\,dx\right).
$$

Choose $g(x)\equiv 1$ we have
$$
\left(\int^1_{-1}f(x)\,dx\right)^2 \le 2\left(\int^1_{-1}|f|^2\,dx\right).
$$

Choose $g(x)=x$ we have
$$
\left(\int^1_{-1}xf(x)\,dx\right)^2 \le \frac{2}{3}\left(\int^1_{-1}|f|^2\,dx\right).
$$

### Example 3

In $\mathbb{M}_{m\times n}$, we define
$$
\langle A, B\rangle = \text{trace}(B^*A),
$$
where the over-$*$ denotes Hermitian transpose (conjugate transpose) as $B^* = \overline{B^T}$.

One can be checked easily that it is indeed an inner product.

The associated norm is defined as
$$
\|A\| = \sqrt{\text{trace}(A^*A)} = \sqrt{\sum_{i,j} |A_{ij}|^2}.
$$
This is the so-called Frobenius norm, and we denote this norm as $\|\cdot\|_F$.

For example,
$$
A = \begin{bmatrix}
1 & 2\\3 & 4\\5 & 6
\end{bmatrix}, \quad \|A\|_F = \sqrt{1^2 + 2^2 + 3^2 + 4^2 + 5^2 + 6^2}.
$$