$$
$$

Lecture 2: Hilbert Spaces

tags: 224a 2020

Last time we defined $L^1(\mathrm{\Omega })$ where $\mathrm{\Omega }\subset {\mathbb{R}}^n$ is a measurable set, and (on the homework) proved that the metric space $(L^1(\mathrm{\Omega }),\Vert \cdot {\Vert }_1)$ is complete. The same thing is true (with a similar proof of completeness) for the $L^p$ norms:
$$\Vert f{\Vert }_p:={\int }_{\mathrm{\Omega }}|f(x){|}^{p}dx,$$
where $1\le p<\mathrm{\infty }$, and the corresponding spaces $L^p(\mathrm{\Omega })$ are caled $L^p$ spaces (some work is required to prove that this actually satisfies the requirements of a norm). These are among the most important examples of Banach Spaces, which are complete normed vector spaces.

Inner Product Spaces

In this lecture we will discuss a more special kind of space which has additional geometric structure similar to that of Euclidean space.

Defn. An inner product space is a (real or complex) vector space $V$ with an inner product $(\cdot ,\cdot ):V\times V\to \mathbb{C}$ satisfying four axioms:
1.$(x,x)\ge 0$
2.$(x,y+z)=(x,y)+(x,z)$
3.$(x,\alpha y)=\alpha (x,y)$ for scalars $\alpha \in \mathbb{C}$ (or $\alpha \in \mathbb{R}$ in the real case).
4.$(x,y)=\overline{(y,x)}$, indicating conjugate linearity in the first coordinate (this just becomes symmetry in the real case).

Some examples of inner product spaces are:
a. The finite dimensional spaces ${\mathbb{R}}^n$ and ${\mathbb{C}}^n$, with the dot product.
b. ${\ell }^2$, the vector space of infinite square summable complex sequences with the inner product
$$(x,y):=\sum _{n=1}^{\mathrm{\infty }}\overline{x_n}{y}_n$$
c. $L^2[a,b]$ with the inner product
$$(f,g):={\int }_a^b\overline{f(x)}g(x)dx$$
d. $V=\{f:[a,b]\to \mathbb{R}:f\text{ is a polynomial}\}$ with the same inner product as above.

Inner product spaces come equipped with a norm, $\Vert x\Vert :=\sqrt{(x,x)}$; with these notions, an orthonormal set is a set of vectors $\{{\varphi }_n{\}}_{n\in A}$ where the index set may be finite or infinite, with the property that $\Vert {\varphi }_n\Vert =1$ for all $n$ and $({\varphi }_n,{\varphi }_m)={\delta }_{mn}$.

The inner product axioms immediately imply the following generalization of Pythagoras' theorem, arguably the defining feature of Euclidean geometry.

Theorem 1. If $\{{\varphi }_n{\}}_{n\le N}$ is a finite orthonormal set and $x$ is any vector in an inner product space, then
$$\Vert x{\Vert }^2=\Vert \sum _{n\le N}({\varphi }_n,x){\varphi }_n{\Vert }^2+\Vert x-(\sum _{n\le N}({\varphi }_n,x){\varphi }_n){\Vert }^2.$$

The proof is just expanding the second term on the RHS using the properties of inner products, and is given in Reed and Simon Thm II.1. As we saw in class, the theorem is false for complex inner product spaces if $({\varphi }_n,x)$ is replaced by $(x,{\varphi }_n)$, which is its conjugate; this subtlety is not present for real inner product spaces. This is perhaps not surprising since $x\mapsto (x,{\varphi }_n)$ is not even a linear map (it is conjugate linear).

The theorem immediately implies the Cauchy-Schwartz inequality (by considering an orthonormal set with one vector), as well as Bessel's inequality:
$$\sum _{n=1}^{\mathrm{\infty }}|({\varphi }_n,x){|}^2\le \Vert x{\Vert }^2,$$
whenever $\{{\varphi }_n\}$ is a (countable) orthonormal sequence in an inner product space (proof: consider the partial sums and apply Theorem 1).

The inner product thereby endows the space with notions of both length (norm) and angle (by taking by $\cos \theta =(x,y)/\Vert x\Vert \Vert y\Vert$). The fact that $\Vert \cdot \Vert$ satisfies the triangle inequality is also a consequence of Cauchy-Schwartz.

Hilbert Spaces

The inner product axioms are enough to reason about finite linear combinations such as those in Theorem 1, but if we want to talk about ``infinite linear combinations'' we need to be able to take limits while staying in the vector space.

Definition. A Hilbert space is a complete inner product space (in the norm induced by the inner product).

Examples a-c mentioned above are Hilbert spaces –- the completness of (b) is on the homework, and the completeness of (c) follows from completeness of $L^2$. Example (d) is not a Hilbert space because it is not complete. One (somewhat overkill) way to see this is to recall the Weierstrass approximation theorem, which says that every continuous real valued function on a closed interval is a uniform limit of polynomials, implying in particular that for any continuous function $f\in C[a,b]$ there is a sequence of polyonmials $p_n\in V\subset C[a,b]$ with $\Vert f-p_n\Vert \to 0$. Taking $f$ to be any continuous function that is not a polynomial, such a sequence of polynomials is Cauchy, but not convergent to any point inside $V$ itself.

The completeness axiom buys us the following useful lemma, the first step towards a generalization of Theorem 1 to the case of countably infinite orthonormal sets.

Lemma 2. If $\{\varphi {\}}_{n=1}^{\mathrm{\infty }}\subset H$ is an orthonormal sequence and $x\in H$ is a vector in a Hilbert space $H$, then there is a vector $y\in H$ (not necessarily equal to $x$) with
$$\sum _{n=1}^{\mathrm{\infty }}({\varphi }_n,x){\varphi }_n=y.$$
An equality of a vector with an infinite series of vectors as above means that the norm of the difference converges to zero along partial sums, i.e., as $N\to \mathrm{\infty }$:
$$\Vert \sum _{n\le N}({\varphi }_n,x){\varphi }_n-y\Vert \to 0.$$
Lemma 2 is proved by observing that the partial sums $S_N:=\sum _{n\le N}({\varphi }_n,x){\varphi }_n$ themselves form a Cauchy sequence due to Bessel's inequality; completeness yields the desired vector $y\in H$.

Remark. The reasoning above also shows that $\sum _{n=1}^{\mathrm{\infty }}{x}_n$ converges as long as $\sum _{n=1}^{\mathrm{\infty }}\Vert x_n\Vert <\mathrm{\infty }.$ This is a good sufficient (but not necessary) condition for checking whether a series with terms in a Hilbert space makes sense.

We are now in a position to define the analogue of an orthonormal basis in infinite dimensions.

Definition. An orthonormal sequence $\{{\varphi }_n{\}}_{n=1}^{\mathrm{\infty }}$ is complete if for every $x\in H$:
$$({\varphi }_n,x)=0\mathrm{\forall }n\Rightarrow x=0,$$
i.e., the only vector orthogonal to every basis vector is $0$. A complete orthonormal sequence is also called an orthonormal basis, which is justified in light of the following theorem showing that it inherits the familiar properties of such bases in ${\mathbb{C}}^n$.

Theorem 3. The following are equivalent:

1. $\{{\varphi }_n\}$ is a complete orthonormal sequence.
2. For every $x\in H$:
   $$x=\sum _{n=1}^{\mathrm{\infty }}({\varphi }_n,x){\varphi }_n.$$
3. For every pair of vectors $x,y\in H$:
   $$(x,y)=\sum _{n=1}^{\mathrm{\infty }}\overline{({\varphi }_n,x)}({\varphi }_n,y).$$
4. (Parseval's Relation)For every $x\in H$:
   $$\Vert x{\Vert }^2=\sum _{n=1}^{\mathrm{\infty }}|({\varphi }_n,x){|}^2.$$

The implications $1\Rightarrow 2$ and $2\Rightarrow 3$ follow from continuity of the inner product, i.e. if $x_n\to x$ and $y_n\to y$ in $H$ then $(x_n,y_n)\to (x,y)$, which can be shown using Cauchy-Schwartz (homework). The remaining implications $3\Rightarrow 4$ and $4\Rightarrow 1$ are immediate. The coefficients $({\varphi }_n,x)$ appearing above are called generalized Fourier coefficients; as we will see later, the coefficients of a Fourier series are a special case of this.

At a high level, linear algebra in infinite dimensions favors orthonormal bases over arbitrary bases because (1) by the reasoning above it is easy to characterize when the "infinite linear combinations" expanding a vector in an orthonormal bases converge, and this is more subtle for arbitrary bases (2) the coefficients in the "infinite linear combination" can be found using inner products, avoiding the need to solve an "infinite system of linear equations". To avoid confusion we will reserve the terms linear combination and span for finite linear combinations only.

We conclude by giving the following nice criterion for the existence of a countable orthonormal basis.

Definition. A metric space is separable if it has a countable dense subset.

Theorem 4. A Hilbert space is separable if and only if it has a countable orthonormal basis.

The proof of the forward direction is via the Gram-Schmidt procedure and is given in Richtmyer Sec 1.6 Theorem 2. The backward direction is left to the homework.

All of the examples given above are separable; we will discuss this in detail in the next lecture.

Remark. It can be shown using Zorn's lemma that every Hilbert space has an orthonormal basis (not necessarily countable). However, uncountable bases never show up in physics, so we will focus on the case of separable spaces.