Chapter 6 extra note 5

Schmidt decomposition
singular value decomposition
image of unit ball in a linear transformation

Selected lecture notes

Let

$T\in \mathcal{L}(V,W)$, where

$V$ and

$W$ are inner product spaces we can define the following operators:

• $T^{\ast }\in \mathcal{L}(W,V)$;
• $T^{\ast }T\in \mathcal{L}(V)$;
• $|T|=\sqrt{T^{\ast }T}\in \mathcal{L}(V)$;
• $T{T}^{\ast }\in \mathcal{L}(W)$;
• $|T^{\ast }|=\sqrt{T{T}^{\ast }}\in \mathcal{L}(W)$;
• $T^{†}\in \mathcal{L}(W,V)$.

Reduced SVD

Since

$T^{\ast }T\ge 0$, we can find

$\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}\subset V$ and

${\lambda }_1\ge \cdots \ge {\lambda }_n\ge 0$ such that

$$\begin{array}{}\text{(1)}& T^{\ast }T{\mathbf{v}}_i={\lambda }_i{\mathbf{v}}_i.\end{array}$$
Define

$r$ be such that

${\lambda }_r>0$ and

${\lambda }_{r+1}=0$, then the singular values of

$T$ are

$\{{\sigma }_i{\}}_{i=1}^r$ where

${\sigma }_i=\sqrt{{\lambda }_i}$. We then have

$$\begin{array}{}\text{(2)}& T^{\ast }T{\mathbf{v}}_i={\sigma }_i^2{\mathbf{v}}_i.\end{array}$$
Let us also define

$$\begin{array}{}\text{(3)}& {\mathbf{w}}_i=\frac{1}{{\sigma }_i}T({\mathbf{v}}_i).\end{array}$$
The reduced singular value decomposition (reduced SVD) of

$T$ is then given by

$$\begin{array}{}\text{(4)}& T\mathbf{v}=\sum _{i=1}^r{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩{\mathbf{w}}_i.\end{array}$$

Remark:

$$\begin{array}{}\text{(5)}& \begin{array}{rl}T{T}^{\ast }{\mathbf{w}}_i& =T{T}^{\ast }(\frac{1}{{\sigma }_i}T({\mathbf{v}}_i))\\ & =\frac{1}{{\sigma }_i}T\left(T^{\ast }T{\mathbf{v}}_i\right)\\ & =\frac{1}{{\sigma }_i}T\left({\sigma }_i^2{\mathbf{v}}_i\right)\\ & ={\sigma }_{i}T{\mathbf{v}}_i={\sigma }_i^2{\mathbf{w}}_i.\end{array}\end{array}$$

Reduced SVD 2

Since

$T{T}^{\ast }\ge 0$, we can find

$\{{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_m\}\subset W$ and

${\lambda }_1\ge \cdots \ge {\lambda }_m\ge 0$ such that

$$\begin{array}{}\text{(6)}& T{T}^{\ast }{\mathbf{w}}_i={\lambda }_i{\mathbf{w}}_i.\end{array}$$
Define

$r$ be such that

${\lambda }_r>0$ and

${\lambda }_{r+1}=0$, then the singular values of

$T$ are

$\{{\sigma }_i{\}}_{i=1}^r$ where

${\sigma }_i=\sqrt{{\lambda }_i}$. We then have

$$\begin{array}{}\text{(7)}& T{T}^{\ast }{\mathbf{w}}_i={\sigma }_i^2{\mathbf{w}}_i.\end{array}$$
Let us also define

$$\begin{array}{}\text{(8)}& {\mathbf{v}}_i=\frac{1}{{\sigma }_i}{T}^{\ast }({\mathbf{w}}_i).\end{array}$$
The reduced singular value decomposition (reduced SVD) of

$T$ is then given by

$$T\mathbf{v}=\sum _{i=1}^r{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩{\mathbf{w}}_i.$$

Remark:
One can either solve the eigenvalue problem for

$T^{\ast }T$ or

$T{T}^{\ast }$ to obtain the singular values and SVD.

---

The adjoint transformation

$$\begin{array}{}\text{(9)}& \begin{array}{rl}⟨\mathbf{v},T^{\ast }\mathbf{w}⟩& =⟨T\mathbf{v},\mathbf{w}⟩\\ & =⟨\sum _{i=1}^r{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩{\mathbf{w}}_i,\mathbf{w}⟩\\ & =\sum _{i=1}^r{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩⟨{\mathbf{w}}_i,\mathbf{w}⟩\\ & =\sum _{i=1}^r{\sigma }_i⟨{\mathbf{w}}_i,\mathbf{w}⟩⟨\mathbf{v},{\mathbf{v}}_i⟩\\ & =⟨\mathbf{v},\sum _{i=1}^r{\sigma }_i\overline{⟨{\mathbf{w}}_i,\mathbf{w}⟩}{\mathbf{v}}_i⟩\\ & =⟨\mathbf{v},\sum _{i=1}^r{\sigma }_i⟨\mathbf{w},{\mathbf{w}}_i⟩{\mathbf{v}}_i⟩.\end{array}\end{array}$$
Therefore, the reduced SVD of

$T^{\ast }$ is given by

$$\begin{array}{}\text{(10)}& T^{\ast }\mathbf{w}=\sum _{i=1}^r{\sigma }_i⟨\mathbf{w},{\mathbf{w}}_i⟩{\mathbf{v}}_i.\end{array}$$

---

Pseudoinverse

The pseudoinverse of

$T$ is given by

$$\begin{array}{}\text{(11)}& T^{†}\mathbf{w}=\sum _{i=1}^r\frac{1}{{\sigma }_i}⟨\mathbf{w},{\mathbf{w}}_i⟩{\mathbf{v}}_i.\end{array}$$

---

Image of the unit ball

Define an unit ball in

$V$ as

$$\begin{array}{}\text{(12)}& B_1=\{\mathbf{v}\in V,\Vert \mathbf{v}\Vert \le 1.\}\end{array}$$
Given

$\mathbf{v}\in B_1$, we have

$$\begin{array}{}\text{(13)}& \mathbf{v}=\sum _{i=1}^n⟨\mathbf{v},{\mathbf{v}}_i⟩{\mathbf{v}}_i,\Vert \mathbf{v}\Vert =\sum _{i=1}^n⟨\mathbf{v},{\mathbf{v}}_i{⟩}^2\le 1.\end{array}$$
Based on SVD, we know that

$$T\mathbf{v}=\sum _{i=1}^r{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩{\mathbf{w}}_i\in W.$$
Let us denote the coordinate of

$T\mathbf{v}$ in the basis

$\{{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_m\}$ as

$(y_1,\cdots ,y_m)$, then we have

$$\begin{array}{}\text{(14)}& y_i=\{\begin{array}{ll}{\sigma }_i⟨\mathbf{v},{\mathbf{v}}_i⟩,& 1\le i\le r\\ \mathbf{0},& r<i\le m.\end{array}\end{array}$$
Therefore

$$\begin{array}{}\text{(15)}& \frac{y_1^2}{{\sigma }_1^2}+\cdots +\frac{y_r^2}{{\sigma }_r^2}=\sum _{i=1}^r⟨\mathbf{v},{\mathbf{v}}_i{⟩}^2\le 1,\end{array}$$
that is,

$T\mathbf{v}$ lies inside the ellipse

$$\begin{array}{}\text{(16)}& E_1=\{\mathbf{w}=(y_1,\cdots ,y_m)\in W,\frac{y_1}{{\sigma }_1^2}+\cdots +\frac{y_r}{{\sigma }_r^2}\le 1.\}\end{array}$$