proof of Gaussian maximizes entropy for given covariance

Let $f,K,\varphi$ be as in the parent (http://planetmath.org/GaussianMaximizesEntropyForGivenCovariance) entry.

The proof uses the nonnegativity of relative entropy $D(f||\varphi )$, and an interesting property of quadratic forms. If $A$ is a quadratic form and $p,q$ are probability distributions each with mean $\mathrm{𝟎}$ and covariance matrix $𝐊$, we have

$$\int p{x}_i{x}_j𝑑x_i𝑑x_j=K_{ij}=\int q{x}_i{x}_j𝑑x_i𝑑x_j$$(1)

and thus

$$\int Ap=\int Aq$$(2)

Now note that since

$$\varphi (𝐱)={\left({(2\pi )}^n|𝐊|\right)}^{-\frac{1}{2}}\exp (-\frac{1}{2}{𝐱}^{\mathrm{T}}{𝐊}^{-1}𝐱),$$(3)

we see that $\log \varphi$ is a quadratic form plus a constant.

0$\le D(f||\varphi )$

$={\displaystyle \int f\log \frac{f}{\varphi }}$

$={\displaystyle \int f\log f}-{\displaystyle \int f\log \varphi }$

$=-h(f)-{\displaystyle \int f\log \varphi }$

$=-h(f)-{\displaystyle \int \varphi \log \varphi }\mathit{\hspace{1em}\hspace{1em}}\text{by the quadratic form property above}$

$=-h(f)+h(\varphi )$

and thus $h(\varphi )\ge h(f)$.