Chapter 5 extra note 1

Selected lecture notes

inner product
norm
Cauchy-Schwartz inequality
Triangular inequality

Inner product space

Definition:
An inner product on a vector space

$V$ is a function that assign to each pair of vectors

$\mathbf{u}\mathbf{,}\mathbf{v}\in V$ a scalar, denoted by

$⟨\mathbf{u},\mathbf{v}⟩$, such that the followings hold: (

$\mathrm{\forall }\mathbf{u},\mathbf{v},\mathbf{w}\in V$,

$\alpha ,\beta \in F$)

1. $⟨\mathbf{u},\mathbf{v}⟩=\overline{⟨\mathbf{v},\mathbf{u}⟩}$, where the overline denotes complex conjugate.
2. $⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩=\alpha ⟨\mathbf{u},\mathbf{w}⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩$.
3. $⟨\mathbf{u},\mathbf{u}⟩\ge 0$.
4. $⟨\mathbf{u},\mathbf{u}⟩=0$ if and only if
   $\mathbf{u}=\mathbf{0}$.

Definition:
A vector space together with an inner product is called an inner product space.

Proposition
(2').

$⟨\mathbf{u},\alpha \mathbf{v}+\beta \mathbf{w}⟩=\overline{\alpha }⟨\mathbf{u},\mathbf{v}⟩+\overline{\beta }⟨\mathbf{u},\mathbf{w}⟩$.

• Proof:

  $$\begin{array}{rl}⟨\mathbf{u},\alpha \mathbf{v}+\beta \mathbf{w}⟩& =\overline{⟨\alpha \mathbf{v}+\beta \mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha ⟨\mathbf{v},\mathbf{u}⟩+\beta ⟨\mathbf{w},\mathbf{u}⟩}\\ & =\overline{\alpha \stackrel{―}{⟨\mathbf{u},\mathbf{v}⟩}+\beta \stackrel{―}{⟨\mathbf{u},\mathbf{w}⟩}}\\ & =\overline{\alpha }⟨\mathbf{u},\mathbf{v}⟩+\overline{\beta }⟨\mathbf{u},\mathbf{w}⟩.\end{array}$$

Definition:
Given an inner product space, we define the norm by

$$\Vert \mathbf{u}\Vert =\sqrt{⟨\mathbf{u},\mathbf{u}⟩}.$$

Theorem: (Cauchy-Schwartz inequality)

$$|⟨\mathbf{u},\mathbf{v}⟩|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert .$$

• Proof:

  If

  $\mathbf{v}=\mathbf{0}$, then

  $⟨\mathbf{u},\mathbf{v}⟩=0$ and

  $\Vert \mathbf{v}\Vert =0$. Done.

  Assume

  $\mathbf{v}\ne \mathbf{0}$, given

  $t\in \mathbb{C}$,
  

  $$\begin{array}{rl}0\le \Vert \mathbf{u}-t\mathbf{v}{\Vert }^2& =⟨\mathbf{u}-t\mathbf{v},\mathbf{u}-t\mathbf{v}⟩\\ & =⟨\mathbf{u},\mathbf{u}⟩-⟨\mathbf{u},t\mathbf{v}⟩-⟨t\mathbf{v},\mathbf{u}⟩+|t{|}^2⟨\mathbf{v},\mathbf{v}⟩\\ & =\Vert \mathbf{u}{\Vert }^2-\overline{t}⟨\mathbf{u},\mathbf{v}⟩-t\overline{⟨\mathbf{u},\mathbf{v}⟩}+|t{|}^2\Vert \mathbf{v}{\Vert }^2.\end{array}$$
  Let

  $t=\frac{⟨\mathbf{u},\mathbf{v}⟩}{\Vert \mathbf{v}{\Vert }^2}$, then
  

  $$\begin{array}{rl}0\le \Vert \mathbf{u}-t\mathbf{v}{\Vert }^2& =\Vert \mathbf{u}{\Vert }^2-2\frac{|⟨\mathbf{u},\mathbf{v}⟩{|}^2}{\Vert \mathbf{v}{\Vert }^2}+\frac{|⟨\mathbf{u},\mathbf{v}⟩{|}^2}{\Vert \mathbf{v}{\Vert }^2}\\ & =\Vert \mathbf{u}{\Vert }^2-\frac{|⟨\mathbf{u},\mathbf{v}⟩{|}^2}{\Vert \mathbf{v}{\Vert }^2}.\end{array}$$
  Therefore,
  

  $$|⟨\mathbf{u},\mathbf{v}⟩|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert .$$

Remark:
The equality of Cauchy-Schwartz inequality holds when

$\Vert \mathbf{u}-t\mathbf{v}\Vert =0$, i.e.,

$\mathbf{u}-t\mathbf{v}=\mathbf{0}$ and

$\mathbf{u}=t\mathbf{v}$. That is, when

$\mathbf{u}$ and

$\mathbf{v}$ are "parallel" to each other.

Lemma:
Let

$V$ be an inner product space, then

$\mathbf{u}=\mathbf{0}$ iff

$⟨\mathbf{u},\mathbf{v}⟩=0$

$\mathrm{\forall }\mathbf{v}\in V$.

• Proof:

  (

  $\Rightarrow$)
  Let

  $\mathbf{u}=\mathbf{0}$, given

  $\mathbf{v}\in V$,
  

  $$⟨\mathbf{u},\mathbf{v}⟩=⟨\mathbf{0},\mathbf{v}⟩=⟨2\cdot \mathbf{0},\mathbf{v}⟩=2⟨\mathbf{0},\mathbf{v}⟩.$$
  So
  

  $$0=2⟨\mathbf{0},\mathbf{v}⟩-⟨\mathbf{0},\mathbf{v}⟩=2⟨\mathbf{0},\mathbf{v}⟩-⟨1\cdot \mathbf{0},\mathbf{v}⟩=2⟨\mathbf{0},\mathbf{v}⟩-1⟨\mathbf{0},\mathbf{v}⟩=1⟨\mathbf{0},\mathbf{v}⟩=⟨\mathbf{0},\mathbf{v}⟩.$$

  (

  $\Leftarrow$)
  Assume

  $⟨\mathbf{u},\mathbf{v}⟩=0$

  $\mathrm{\forall }\mathbf{v}\in V$. Choose

  $\mathbf{v}=\mathbf{u}$ we have
  

  $$⟨\mathbf{u},\mathbf{u}⟩=0.$$
  Based on condition 4 of the inner product space, we have

  $\mathbf{u}=\mathbf{0}$.

Corollary:
Let

$\mathbf{u},\mathbf{v}\in V$ where

$V$ is an inner product space, then

$\mathbf{u}=\mathbf{v}$ iff

$⟨\mathbf{u},\mathbf{z}⟩=⟨\mathbf{v},\mathbf{z}⟩$

$\mathrm{\forall }\mathbf{z}\in V$.

Lemma: (Triangular inequality)

$\mathrm{\forall }\mathbf{u}\mathbf{,}\mathbf{v}\in V$ where

$V$ is an inner product space, we have

$$\Vert \mathbf{u}+\mathbf{v}\Vert \le \Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert .$$

• Proof:

  $$\begin{array}{rl}\Vert \mathbf{u}+\mathbf{v}{\Vert }^2& =⟨\mathbf{u}+\mathbf{v},\mathbf{u}+\mathbf{v}⟩\\ & =\Vert \mathbf{u}{\Vert }^2+⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{v},\mathbf{u}⟩+\Vert \mathbf{v}{\Vert }^2\\ & =\Vert \mathbf{u}{\Vert }^2+2\text{Re}⟨\mathbf{u},\mathbf{v}⟩+\Vert \mathbf{v}{\Vert }^2\\ & \le \Vert \mathbf{u}{\Vert }^2+2|⟨\mathbf{u},\mathbf{v}⟩|+\Vert \mathbf{v}{\Vert }^2\\ & \le \Vert \mathbf{u}{\Vert }^2+2\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert +\Vert \mathbf{v}{\Vert }^2\\ & ={(\Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert )}^2.\end{array}$$

Examples

Example 1

In

${\mathbb{R}}^2$, show that the following defines an inner product

$$⟨(x_1,x_2),(y_1,y_2)⟩=x_1{y}_1+2x_2{y}_2.$$

• Proof:

  (1)

  $⟨(x_1,x_2),(y_1,y_2)⟩=x_1{y}_1+2x_2{y}_2=⟨(y_1,y_2),(x_1,x_2)⟩$.
  (2)
  

  $$\begin{array}{rl}⟨\alpha (x_1,x_2)+\beta (y_1,y_2),(z_1,z_2)⟩& =(\alpha x_1+\beta y_1)z_1+2(\alpha x_2+\beta y_2)z_2\\ & =\alpha (x_1{z}_1+2x_2{z}_2)+\beta (y_1{z}_1+2y_2{z}_2)\\ & =\alpha ⟨(x_1,x_2),(z_1,z_2)⟩+\beta ⟨(y_1,y_2),(z_1,z_2)⟩.\end{array}$$
  (3)

  $⟨(x_1,x_2),(x_1,x_2)⟩=x_1^2+2x_2^2\ge 0$.
  (4) If

  $⟨(x_1,x_2),(x_1,x_2)⟩=x_1^2+2x_2^2=0$, then

  $x_1=x_2=0$, i.e.,

  $(x_1,x_2)=(0,0)$.

With this inner product, the norm is defined as

$$\Vert (x_1,x_2)\Vert =\sqrt{x_1^2+2x_2^2}.$$

We can then apply Cauchy Schwartz inequality to have

$$(x_1{y}_1+2x_2{y}_2{)}^2=|⟨(x_1,x_2),(y_1,y_2)⟩{|}^2\le \Vert (x_1,x_2){\Vert }^2\Vert (y_1,y_2){\Vert }^2=(x_1^2+2x_2^2)(y_1^2+2y_2^2).$$

Example 2

In

${\mathbb{P}}_n([-1,1])$, we can define the inner product as

$$⟨f(x),g(x)⟩={\int }_{-1}^{1}f(x)g(x)dx.$$
If we allow polynomials with complex coefficients, the inner product should be defined as

$$⟨f(x),g(x)⟩={\int }_{-1}^{1}f(x)\overline{g(x)}dx.$$

The norm is then defined by

$$\Vert f(x)\Vert =\sqrt{{\int }_{-1}^1|f(x){|}^{2}dx},$$
and the Cauchy Schwartz inequality reads

$${({\int }_{-1}^{1}f(x)\overline{g(x)}dx)}^2=|⟨f,g⟩{|}^2\le \Vert f{\Vert }^2\Vert g{\Vert }^2=\left({\int }_{-1}^1|f{|}^{2}dx\right)\left({\int }_{-1}^1|g{|}^{2}dx\right).$$

Choose

$g(x)\equiv 1$ we have

$${({\int }_{-1}^{1}f(x)dx)}^2\le 2\left({\int }_{-1}^1|f{|}^{2}dx\right).$$

Choose

$g(x)=x$ we have

$${({\int }_{-1}^{1}xf(x)dx)}^2\le \frac{2}{3}\left({\int }_{-1}^1|f{|}^{2}dx\right).$$

Example 3

In

${\mathbb{M}}_{m\times n}$, we define

$$⟨A,B⟩=\text{trace}(B^{\ast }A),$$
where the over-

$\ast$ denotes Hermitian transpose (conjugate transpose) as

$B^{\ast }=\overline{B^T}$.

One can be checked easily that it is indeed an inner product.

The associated norm is defined as

$$\Vert A\Vert =\sqrt{\text{trace}(A^{\ast }A)}=\sqrt{\sum _{i,j}|A_{ij}{|}^2}.$$
This is the so-called Frobenius norm, and we denote this norm as

$\Vert \cdot {\Vert }_F$.

For example,

$$A=\left[\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right],\Vert A{\Vert }_F=\sqrt{1^2+2^2+3^2+4^2+5^2+6^2}.$$