Chapter 5 extra note 5

Selected lecture notes

Rank–Nullity Theorem
Four fundamental subspaces
least squares solutions
solvability condition

Rank–nullity theorem

Remarks:
It should be clear that, given a linear transformation

$T:V\to W$,

$$\text{Ker}(T)\oplus \text{Ker}(T{)}^{\perp }=V,\text{Ran}(T)\oplus \text{Ran}(T{)}^{\perp }=W.$$

Proposition:
Let

$V=W_1\oplus W_2$, where

$W_1$ and

$W_2$ are finite-dimensional spaces, then

$$\text{dim}(V)=\text{dim}(W_1)+\text{dim}(W_2).$$

• Proof:

  (Hint of the proof)
  Let

  $\{{\mathbf{u}}_1,\cdots ,{\mathbf{u}}_m\}$ be a basis of

  $W_1$ and

  $\{{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ be a basis of

  $W_2$.
  Then show that

  $\{{\mathbf{u}}_1,\cdots ,{\mathbf{u}}_m,{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n\}$ is a basis of

  $V$.

Proposition:
Let

$T:V\to W$ be linear, then

$$\text{dim}(V)=\text{dim}(\text{Ker}(T))+\text{dim}(\text{Ker}(T{)}^{\perp }),$$
and

$$\text{dim}(W)=\text{dim}(\text{Ran}(T))+\text{dim}(\text{Ran}(T{)}^{\perp }).$$

The 'essential' part of a linear transformation

Let us define a map that is a restriction of

$T$ to the domain

$\text{Ker}(T{)}^{\perp }$ and its range

$\text{Ran}(T)$:

$$\tilde{T}:\text{Ker}(T{)}^{\perp }\subseteq V\to \text{Ran}(T)\subseteq W.$$

Lemma:

$\tilde{T}$ is an isomorphism.

• Proof:

  (linearity) Trivial.

  (1-1) claim:

  $\text{Ker}(\tilde{T})=\{\mathbf{0}\}$.

  Since

  $\text{Ker}(\tilde{T})$ is a vector subspace,

  $\mathbf{0}\in \text{Ker}(\tilde{T})$.
  So

  $\{\mathbf{0}\}\subseteq \text{Ker}(\tilde{T})$.

  Suppose

  $\mathbf{u}\in \text{Ker}(\tilde{T})$, then we must have

  $\mathbf{u}\in \text{Ker}(T{)}^{\perp }$ and

  $\tilde{T}(\mathbf{u})=\mathbf{0}$.
  So,

  $T(\mathbf{u})=\mathbf{0}$ and hence

  $\mathbf{u}\in \text{Ker}(T)$.
  Therefore,

  $\mathbf{u}\in (\text{Ker}(T{)}^{\perp }\cap \text{Ker}(T))=\{\mathbf{0}\}$.
  That gives

  $\text{Ker}(\tilde{T})\subseteq \{\mathbf{0}\}$.

  (onto)
  claim: Given

  $\mathbf{w}\in \text{Ran}(T)$, there exists a

  $\mathbf{u}\in \text{Ker}(T{)}^{\perp }$ such that

  $\tilde{T}(\mathbf{u})=\mathbf{w}$.

  Given

  $\mathbf{w}\in \text{Ran}(T)$, there exists

  $\mathbf{v}\in V$ such that

  $T(\mathbf{v})=\mathbf{w}$.
  Since

  $\mathbf{v}\in V$, there exists

  $\mathbf{u}\in \text{Ker}(T{)}^{\perp }$ and

  ${\mathbf{v}}_k\in \text{Ker}(T)$ such that

  $\mathbf{v}=\mathbf{u}+{\mathbf{v}}_k$.
  We then have
  

  $$\mathbf{w}=T(\mathbf{v})=T(\mathbf{u}+{\mathbf{v}}_k)=T(\mathbf{u})+T({\mathbf{v}}_k)=T(\mathbf{u})=\tilde{T}(\mathbf{u}).$$

Corollary:

1. $\text{Ker}(T{)}^{\perp }$ and
   $\text{Ran}(T)$ are isomorphic.
2. $\text{dim}(\text{Ker}(T{)}^{\perp })=\text{dim}(\text{Ran}(T))$.

Rank–Nullity Theorem:
Let

$T:V\to W$ be linear, then

$$\text{dim}(V)=\text{dim}(\text{Ker}(T))+\text{dim}(\text{Ran}(T)).$$

• Proof:

  $$\begin{array}{rl}\text{dim}(V)& =\text{dim}(\text{Ker}(T))+\text{dim}(\text{Ker}(T{)}^{\perp })\\ & =\text{dim}(\text{Ker}(T))+\text{dim}(\text{Ran}(T)).\end{array}$$

Four fundamental subspaces

Theorem:

1. $\text{Ker}(T^{\ast })=\text{Ran}(T{)}^{\perp }$.
   • Proof:

     Let

     $\mathbf{x}\in \text{Ran}(T{)}^{\perp }$.
     

     $\iff$

     $⟨\mathbf{x},\mathbf{w}⟩=0,\mathrm{\forall }\mathbf{w}\in \text{Ran}(T)$.
     

     $\iff$

     $⟨\mathbf{x},T(\mathbf{v})⟩=0,\mathrm{\forall }\mathbf{v}\in V$.
     

     $\iff$

     $⟨T^{\ast }(\mathbf{x}),\mathbf{v}⟩=0,\mathrm{\forall }\mathbf{v}\in V$.
     

     $\iff$

     $T^{\ast }(\mathbf{x})=\mathbf{0}$.
     

     $\iff$

     $\mathbf{x}\in \text{Ker}(T^{\ast })$.

2. $\text{Ker}(T)=\text{Ran}(T^{\ast }{)}^{\perp }$.
3. $\text{Ran}(T)=\text{Ker}(T^{\ast }{)}^{\perp }$.
4. $\text{Ran}(T^{\ast })=\text{Ker}(T{)}^{\perp }$.

Corollary:
Let

$T:V\to W$ be linear, then

$$\text{Ker}(T)\oplus \text{Ran}(T^{\ast })=V,\text{Ran}(T)\oplus \text{Ker}(T^{\ast })=W.$$

Least squares solution

Let

$T:V\to W$ be a linear transformation, we consider solving the problem

$T(\mathbf{v})=\mathbf{w}$, where

$\mathbf{w}$ is a given vector.

• If
  $\mathbf{w}\in \text{Ran}(T)$, there exists solutions.
  • If
    $\mathbf{w}\in \text{Ran}(T)$ and
    $\text{Ker}(T)=\mathbf{0}$, there exists an unique solution.
• If
  $\mathbf{w}\notin \text{Ran}(T)$, there does not exist any solution.
  • We can define the least squares solution as
    
    $${\mathbf{v}}^{\ast }=\text{arg}\underset{\mathbf{v}\in V}{min}\Vert T(\mathbf{v})-\mathbf{w}{\Vert }_2^2.$$
    • The minimum value is achieved when
      $\mathbf{w}$ is projected to
      $\text{Ran}(T)$, and hence the least squares solution satisfies
      
      $$T({\mathbf{v}}^{\ast })=P_{\text{Ran}(T)}\mathbf{w}.$$
    • The minimum value is achieved when
      $\mathbf{w}$ is projected to
      $\text{Ran}(T)$, and hence
      
      $$(\mathbf{w}-T({\mathbf{v}}^{\ast }))\perp \text{Ran}(T).$$
      In other words,
      
      $$⟨\mathbf{w}-T({\mathbf{v}}^{\ast }),T(\mathbf{v})⟩=0,\mathrm{\forall }\mathbf{v}\in V.$$
      We can also rewrite it as
      
      $$⟨T({\mathbf{v}}^{\ast }),T(\mathbf{v})⟩=⟨\mathbf{w},T(\mathbf{v})⟩,\mathrm{\forall }\mathbf{v}\in V.$$
      Use the adjoint transformation we obtain
      
      $$⟨T^{\ast }T({\mathbf{v}}^{\ast }),\mathbf{v}⟩=⟨T^{\ast }(\mathbf{w}),\mathbf{v}⟩,\mathrm{\forall }\mathbf{v}\in V.$$
      That is,
      
      $$T^{\ast }T({\mathbf{v}}^{\ast })=T^{\ast }(\mathbf{w}),$$
      which is often called the normal equation for the least squares solutions.

Solvability condition

Notice that

$\text{Ran}(T{)}^{\perp }=\text{Ker}(T^{\ast })$, which means that if

$\mathbf{w}\in \text{Ran}(T)$, then

$\mathbf{w}\perp \text{Ker}(T^{\ast })$. We then obtain the solvability condition as

$$⟨\mathbf{w},\mathbf{u}⟩=0,\mathrm{\forall }\mathbf{u}\in \text{Ker}(T^{\ast }).$$