Rayleigh-Ritz theorem

Let $A\in {𝐂}^{n\times n}$ be a Hermitian matrix. Then itseigenvectors are the critical points (vectors) of the ”Rayleigh quotient”,which is the real function $R:{ℂ}^n\backslash \backslash \{\mathrm{𝟎}\}\to ℝ$

$$R(𝐱)=\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱},\parallel 𝐱\parallel \ne 0$$

and its eigenvalues are its values at such critical points.

As a consequence, we have:

$${\lambda }_{max}=\underset{\parallel 𝐱\parallel \ne 0}{\max }\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$$

and

$${\lambda }_{min}=\underset{\parallel 𝐱\parallel \ne 0}{\min }\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$$

Proof:

First of all, let’s observe that for a hermitian matrix, the number ${𝐱}^{H}A𝐱$ is a real one (actually, ,whence is real), so that theRayleigh quotient is real as well.

Let’s now compute the critical points $\overline{𝐱}$ of theRayleigh quotient, i.e. let’s solve the equations system $\frac{dR(\overline{𝐱})}{d𝐱}={\mathrm{𝟎}}^T$. Let’s write $𝐱={𝐱}^{(R)}+j{𝐱}^{(I)}$, ${𝐱}^{(R)}$ and ${𝐱}^{(I)}$ beingrespectively the real and imaginary part of $𝐱$. We have:

$$\frac{dR(𝐱)}{d𝐱}=\frac{dR(𝐱)}{d{𝐱}^{(R)}}+j\frac{dR(𝐱)}{d{𝐱}^{(I)}}$$

so that we must have:

$$\frac{dR(\overline{𝐱})}{d{𝐱}^{(R)}}=\frac{dR(\overline{𝐱})}{d{𝐱}^{(I)}}={\mathrm{𝟎}}^T$$

Using derivatives rules, we obtain:

${\displaystyle \frac{dR(𝐱)}{d{𝐱}^{(R)}}}$$=$${\displaystyle \frac{d}{d{𝐱}^{(R)}}}\left({\displaystyle \frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}}\right)={\displaystyle \frac{\frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(R)}}{𝐱}^H𝐱-{𝐱}^{H}A𝐱\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(R)}}}{{\left({𝐱}^H𝐱\right)}^2}}={\displaystyle \frac{\frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(R)}}-R(𝐱)\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(R)}}}{{𝐱}^H𝐱}}\text{.}$

Applying matrix calculus rules, we find:

${\displaystyle \frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(R)}}}$$=$${𝐱}^{H}A{\displaystyle \frac{d𝐱}{d{𝐱}^{(R)}}}+{𝐱}^T{A}^T{\displaystyle \frac{d{𝐱}^{\ast }}{d{𝐱}^{(R)}}}={𝐱}^{H}A+{𝐱}^T{A}^T={𝐱}^{H}A+{\left({𝐱}^H{A}^H\right)}^{\ast }=$

and since $A=A^H$,

$$={𝐱}^{H}A+{\left({𝐱}^{H}A\right)}^{\ast }=2\mathrm{\Re }\left({𝐱}^{H}A\right)\text{.}$$

In a similar way, we get:

$$\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(R)}}=2\mathrm{\Re }\left({𝐱}^H\right)\text{.}$$

Substituting, we obtain:

$$\frac{dR(𝐱)}{d{𝐱}^{(R)}}=2\frac{\mathrm{\Re }\left({𝐱}^{H}A\right)-R(𝐱)\mathrm{\Re }\left({𝐱}^H\right)}{{𝐱}^H𝐱}$$

and, after a transposition, equating to the null column vector,

$\mathrm{𝟎}$$=$${\left(\mathrm{\Re }\left({\overline{𝐱}}^{H}A\right)-R(\overline{𝐱})\mathrm{\Re }\left({\overline{𝐱}}^H\right)\right)}^T=$

$=$$\mathrm{\Re }\left(A^T{\overline{𝐱}}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Re }\left({\overline{𝐱}}^{\ast }\right)=\mathrm{\Re }\left({(A^H\overline{𝐱})}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Re }\left({\overline{𝐱}}^{\ast }\right)=$

$=$$\mathrm{\Re }\left({(A\overline{𝐱})}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Re }\left({\overline{𝐱}}^{\ast }\right)=\mathrm{\Re }(A\overline{𝐱})-R(\overline{𝐱})\mathrm{\Re }\left(\overline{𝐱}\right)$

and, since $R(𝐱)$ is real,

$$\mathrm{\Re }(A\overline{𝐱}-R(\overline{𝐱})\overline{𝐱})=\mathrm{𝟎}$$

Let’s then evaluate $\frac{dR(𝐱)}{d{𝐱}^{(I)}}$:

${\displaystyle \frac{dR(𝐱)}{d{𝐱}^{(I)}}}$$=$${\displaystyle \frac{d}{d{𝐱}^{(I)}}}\left({\displaystyle \frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}}\right)={\displaystyle \frac{\frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(I)}}{𝐱}^H𝐱-{𝐱}^{H}A𝐱\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(I)}}}{{\left({𝐱}^H𝐱\right)}^2}}={\displaystyle \frac{\frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(I)}}-R(𝐱)\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(I)}}}{{𝐱}^H𝐱}}\text{.}$

Applying again matrix calculus rules, we find:

${\displaystyle \frac{d\left({𝐱}^{H}A𝐱\right)}{d{𝐱}^{(I)}}}$$=$${𝐱}^{H}A{\displaystyle \frac{d𝐱}{d{𝐱}^{(I)}}}+{𝐱}^T{A}^T{\displaystyle \frac{d{𝐱}^{\ast }}{d{𝐱}^{(I)}}}=j{𝐱}^{H}A-j{𝐱}^T{A}^T=j({𝐱}^{H}A-{\left({𝐱}^H{A}^H\right)}^{\ast })=$

and since $A=A^H$,

$$=j({𝐱}^{H}A-{\left({𝐱}^{H}A\right)}^{\ast })=j(2j\mathrm{\Im }\left({𝐱}^{H}A\right))=-2\mathrm{\Im }\left({𝐱}^{H}A\right)\text{.}$$

In a similar way, we get:

$$\frac{d\left({𝐱}^H𝐱\right)}{d{𝐱}^{(I)}}=j{𝐱}^H-j{𝐱}^T=j({𝐱}^H-{\left({𝐱}^H\right)}^{\ast })=j(2j\mathrm{\Im }({𝐱}^H))=-2\mathrm{\Im }\left({𝐱}^H\right)\text{.}$$

Substituting, we obtain:

$$\frac{dR(𝐱)}{d{𝐱}^{(I)}}=-2\frac{\mathrm{\Im }\left({𝐱}^{H}A\right)-R(𝐱)\mathrm{\Im }\left({𝐱}^H\right)}{{𝐱}^H𝐱}$$

and, after a transposition, equating to the null column vector,

$\mathrm{𝟎}$$=$${\left(\mathrm{\Im }\left({\overline{𝐱}}^{H}A\right)-R(\overline{𝐱})\mathrm{\Im }\left({\overline{𝐱}}^H\right)\right)}^T=$

$=$$\mathrm{\Im }\left(A^T{\overline{𝐱}}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Im }\left({\overline{𝐱}}^{\ast }\right)=\mathrm{\Im }\left({(A^H\overline{𝐱})}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Im }\left({\overline{𝐱}}^{\ast }\right)=$

$=$$\mathrm{\Im }\left({(A\overline{𝐱})}^{\ast }\right)-R(\overline{𝐱})\mathrm{\Im }\left({\overline{𝐱}}^{\ast }\right)=-\mathrm{\Im }(A\overline{𝐱})+R(\overline{𝐱})\mathrm{\Im }\left(\overline{𝐱}\right)$

and, since $R(𝐱)$ is real,

$$\mathrm{\Im }(A\overline{𝐱}-R(\overline{𝐱})\overline{𝐱})=\mathrm{𝟎}$$

In conclusion, we have that a stationary vector $\overline{𝐱}$ forthe Rayleigh quotient satisfies the complex eigenvalue equation

$$A\overline{𝐱}-R(\overline{𝐱})\overline{𝐱}=\mathrm{𝟎}$$

whence the thesis.$\mathrm{\square }$

Remarks:

1) The two relations ${\lambda }_{\max }={\max }_{\parallel 𝐱\parallel \ne 0}\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$ and ${\lambda }_{\min }={\min }_{\parallel 𝐱\parallel \ne \mathrm{𝟎}}\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$ can also be obtained in a simpler way.By Schur’s canonical form theorem, any normal (and hence any hermitian)matrix is unitarily diagonalizable, i.e. a unitary matrix $U$ exists suchthat $A=U\mathrm{\Lambda }{U}^H$ with $\mathrm{\Lambda }=diag({\lambda }_1,{\lambda }_2,\mathrm{\cdots },{\lambda }_n)$. So, since all eigenvalues of a hermitian matrix arereal, it’s possible to write:

${𝐱}^{H}A𝐱$$=𝐱$${}^{H}U\mathrm{\Lambda }{U}^H𝐱={\left(U^H𝐱\right)}^H\mathrm{\Lambda }\left(U^H𝐱\right)={𝐲}^H\mathrm{\Lambda }𝐲={\displaystyle \sum _{i=1}^n}{\lambda }_i{\left|y_i\right|}^2\le {\lambda }_{\max }{\displaystyle \sum _{i=1}^n}{\left|y_i\right|}^2=$

$=$${\lambda }_{\max }{𝐲}^H𝐲={\lambda }_{\max }{\left(U^H𝐱\right)}^H\left(U^H𝐱\right)={\lambda }_{\max }\left({𝐱}^{H}U{U}^H𝐱\right)={\lambda }_{\max }\left({𝐱}^H𝐱\right)$

whence

$${\lambda }_{\max }\ge \frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$$

But, having defined $A{𝐯}_M={\lambda }_{\max }{𝐯}_M$, wehave:

$$\frac{{𝐯}_M^{H}A{𝐯}_M}{{𝐯}_M^H{𝐯}_M}=\frac{{𝐯}_M^H{\lambda }_{\max }{𝐯}_M}{{𝐯}_M^H{𝐯}_M}={\lambda }_{\max }$$

so that

$${\lambda }_{\max }=\underset{\parallel 𝐱\parallel \ne \mathrm{𝟎}}{\max }\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$$

In a much similar way, we obtain

$${\lambda }_{\min }=\underset{\parallel 𝐱\parallel \ne \mathrm{𝟎}}{\min }\frac{{𝐱}^{H}A𝐱}{{𝐱}^H𝐱}$$

2) The above relations yield the following noteworthing bounds for thediagonal entries of a hermitian matrix:

$${\lambda }_{\min }\le a_{ii}\le {\lambda }_{\max }$$

In fact, having defined

$${𝐞}_i={[\stackrel{i-1}{\overbrace{0,0,\mathrm{\dots },0,}}1,0,\mathrm{\dots },0]}^T$$

and observing that $a_{ii}=\frac{{𝐞}_i^{H}A{𝐞}_i}{{𝐞}_i^H{𝐞}_i}=R({𝐞}_i)$, we have:

$${\lambda }_{\min }=\underset{\parallel 𝐱\parallel \ne \mathrm{𝟎}}{\min }R(𝐱)\le R({𝐞}_i)\le \underset{\parallel 𝐱\parallel \ne \mathrm{𝟎}}{\max }R(𝐱)={\lambda }_{\max }\text{.}$$