$$

Rayleigh quotient of a real symmetric matrix

Problem

Let

$A$ be an

$n\times n$ real symmetric matrix with its eigenvalues

${\lambda }_1\le \cdots \le {\lambda }_n$. Show that for any

$\mathbf{x}\in {\mathbb{R}}^n$ with

$\Vert \mathbf{x}\Vert =1$, we have

$${\lambda }_1\le {\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}\le {\lambda }_n.$$

Moreover, when

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}={\lambda }_1$,

$\mathbf{x}$ is an eigenvector of

$A$ with respect to

${\lambda }_1$; when

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}={\lambda }_n$,

$\mathbf{x}$ is an eigenvector of

$A$ with respect to

${\lambda }_n$.

Thought

According to the spectral theorem, a real symmetric matrix is similar to a diagonal matrix by an orthonormal basis. That is, there is an orthonormal basis

$\beta$ such that the matrix

$A$, when observed by

$\beta$, has its matrix representation diagonal. Intuitively, the proof should be similar to the case when

$A$ is a diagonal matrix.

Sample answer

By the spectral theorem, there is an orthonormal basis

$\beta =\{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_n\}$ composed of eigenvectors of

$A$. We may assume that

${\mathbf{u}}_i$ is the eigenvector corresponding to

${\lambda }_i$ for

$i=1,\dots ,n$.

Since

$\beta$ is an orthonormal basis of

${\mathbb{R}}^n$, we may write

$$\mathbf{x}=c_1{\mathbf{u}}_1+\cdots +c_n{\mathbf{u}}_n$$

for some scalars

$c_1,\dots ,c_n$. As

${\mathbf{u}}_i$'s are eigenvectors, we also have

$$A\mathbf{x}=c_1{\lambda }_1{\mathbf{u}}_1+\cdots +c_n{\lambda }_n{\mathbf{u}}_n.$$

The fact that

$\beta$ is orthonormal means that

$⟨{\mathbf{u}}_i,{\mathbf{u}}_j⟩=0$ for

$i\ne j$ and

$⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩=1$ for any

$i$. By the distributive law of the inner product, we may compute

$$\begin{array}{rl}{\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}& =⟨\mathbf{x},A\mathbf{x}⟩\\ & =⟨c_1{\mathbf{u}}_1+\cdots +c_n{\mathbf{u}}_n,c_1{\lambda }_1{\mathbf{u}}_1+\cdots +c_n{\lambda }_n{\mathbf{u}}_n⟩\\ & =\sum _{i=1}^n{c}_i^2{\lambda }_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩+\sum _{i\ne j}{c}_i{c}_j{\lambda }_j⟨{\mathbf{u}}_i,{\mathbf{u}}_j⟩\\ & =c_1^2{\lambda }_1+\cdots +c_n^2{\lambda }_n\end{array}$$

and

$$\begin{array}{rl}\Vert \mathbf{x}{\Vert }^2& =⟨\mathbf{x},\mathbf{x}⟩\\ & =⟨c_1{\mathbf{u}}_1+\cdots +c_n{\mathbf{u}}_n,c_1{\mathbf{u}}_1+\cdots +c_n{\mathbf{u}}_n⟩\\ & =\sum _{i=1}^n{c}_i^2⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩+\sum _{i\ne j}{c}_i{c}_j⟨{\mathbf{u}}_i,{\mathbf{u}}_j⟩\\ & =c_1^2+\cdots +c_n^2.\end{array}$$

By replacing

$c_i^2=p_i$ for

$i=1,\dots ,n$, we have

$$\begin{array}{rl}{\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}& =p_1\cdot {\lambda }_1+\cdots +p_n\cdot {\lambda }_n,\\ \Vert \mathbf{x}{\Vert }^2& =p_1+\cdots +p_n=1,\end{array}$$

and

$p_i\ge 0$ for

$i=1,\dots ,n$. Thus,

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}$ is the weighted average of

${\lambda }_1,\dots ,{\lambda }_n$ with respect to the weights

$p_1,\dots ,p_n$. Consequently, the value of

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}$ is between the minimum and the maximum of

$\{{\lambda }_1,\dots ,{\lambda }_n\}$, which are

${\lambda }_1$ and

${\lambda }_n$, respectively. When

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}={\lambda }_1$, necessarily all the weight is attributed to those

${\lambda }_i$ with

${\lambda }_i=1$, and

$\mathbf{x}$ is a linear combination of eigenvectors with respect to

${\lambda }_1$, which is again an eigenvector with respect to

${\lambda }_1$. Similarly, when

${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}={\lambda }_n$,

$\mathbf{x}$ is an eigenvector with respect to

${\lambda }_n$.

This note can be found at Course website > Learning resources.