differential entropy

Let $(X,𝔅,\mu )$ be a probability space, and let $f\in L^p(X,𝔅,\mu )$, ${||f||}_p=1$ be a function. The differential entropy $h(f)$ is defined as

$$h(f)\equiv -{\int }_X{|f|}^p\log {|f|}^{p}d\mu$$(1)

Differential entropy is the continuous version of the Shannon entropy, $H[𝐩]=-{\sum }_i{p}_i\log p_i$. Consider first $u_a$, the uniform 1-dimensional distribution on $(0,a)$. The differential entropy is

$$h(u_a)=-{\int }_0^a\frac{1}{a}\log \frac{1}{a}d\mu =\log a.$$(2)

Next consider probability distributions such as the function

$$g=\frac{1}{2\pi \sigma }{e}^{-\frac{{(t-\mu )}^2}{2{\sigma }^2}},$$(3)

the 1-dimensional Gaussian. This pdf has differential entropy

$$h(g)=-{\int }_{ℝ}g\log gdt=\frac{1}{2}\log 2\pi e{\sigma }^2.$$(4)

For a general $n$-dimensional Gaussian (http://planetmath.org/JointNormalDistribution) ${𝒩}_n(\mu ,𝐊)$ with mean vector $\mu$ and covariance matrix $𝐊$, $K_{ij}=\mathrm{cov}(x_i,x_j)$, we have

$$h({𝒩}_n(\mu ,𝐊))=\frac{1}{2}\log {(2\pi e)}^n|𝐊|$$(5)

where $|𝐊|=det𝐊$.