exponential family

A probability (density) function $f_X(x\mid \theta )$ given a parameter $\theta$ is said to belong to the (one parameter) exponential family of distributions if it can be written in one of the following two equivalent forms:

1. $a(x)b(\theta )\exp \left[c(x)d(\theta )\right]$

2. $\exp \left[a(x)+b(\theta )+c(x)d(\theta )\right]$

where $a,b,c,d$ are known functions.If $c(x)=x$, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta )$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.

Examples:

• The normal distribution, $N(\mu ,{\sigma }^2)$, treating ${\sigma }^2$ as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithm of $N(\mu ,{\sigma }^2)$ to get

  $$-\frac{1}{2}\ln (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}{(x-\mu )}^2$$

  Rearrange the above expression and we have

  $$\frac{x\mu }{{\sigma }^2}-\frac{{\mu }^2}{2{\sigma }^2}-\frac{1}{2}\left[\frac{x^2}{{\sigma }^2}+\ln (2\pi {\sigma }^2)\right]$$

  Set $c(x)=x$, $d(\mu )=\mu /{\sigma }^2$, $b(\mu )=-{\mu }^2/(2{\sigma }^2)$, and$a(x)=-1/2\left[x^2/{\sigma }^2+\ln (2\pi {\sigma }^2)\right]$. Then we see that $N(\mu ,{\sigma }^2)$ does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is $d(\mu )=\mu /{\sigma }^2$.

• Similarly, the Poisson, binomial, Gamma, and inverse Gaussian distributions all belong to the exponential family and they are all in canonical form.

• Lognormal and Weibull distributions also belong to the exponential family but they are not in canonical form.

Remarks

• If the p.d.f of a random variable $X$ belongs to an exponential family, and it is expressed in the second of the two above forms, then

  $$\mathrm{E}[c(X)]=-\frac{b^{\prime }(\theta )}{d^{\prime }(\theta )},$$(1)

  and

  $$\mathrm{Var}[c(X)]=\frac{d^{\prime \prime }(\theta )b^{\prime }(\theta )-d^{\prime }(\theta )b^{\prime \prime }(\theta )}{d^{\prime }{(\theta )}^3},$$(2)

  provided that functions $b$ and $d$ are appropriately conditioned.

• Given a member from the exponential family of distributions, we have$\mathrm{E}[U]=0$ and $I=-\mathrm{E}[U^{\prime }]$, where $U$ is the score function and $I$ the Fisher information. To see this, first observe that the log-likelihood function from a member of the exponential family of distributions is given by

  $$\mathrm{\ell }(\theta \mid x)=a(x)+b(\theta )+c(x)d(\theta ),$$

  and hence the score function is

  $$U(\theta )=b^{\prime }(\theta )+c(X)d^{\prime }(\theta ).$$

  From (1), $\mathrm{E}[U]=0$.Next, we obtain the Fisher information $I$. By definition, we have

  $I$$=$$\mathrm{E}[U^2]-\mathrm{E}{[U]}^2$

  $=$$\mathrm{E}[U^2]$

  $=$$d^{\prime }{(\theta )}^2\mathrm{Var}[c(X)]$

  $=$${\displaystyle \frac{d^{\prime \prime }(\theta )b^{\prime }(\theta )-d^{\prime }(\theta )b^{\prime \prime }(\theta )}{d^{\prime }(\theta )}}$

  On the other hand,

  $$\frac{\partial U}{\partial \theta }=b^{\prime \prime }(\theta )+c(X)d^{\prime \prime }(\theta )$$

  so

  $\mathrm{E}\left[{\displaystyle \frac{\partial U}{\partial \theta }}\right]$$=$$b^{\prime \prime }(\theta )+\mathrm{E}[c(X)]d^{\prime \prime }(\theta )$

  $=$$b^{\prime \prime }(\theta )-{\displaystyle \frac{b^{\prime }(\theta )}{d^{\prime }(\theta )}}{d}^{\prime \prime }(\theta )$

  $=$${\displaystyle \frac{b^{\prime \prime }(\theta )d^{\prime }(\theta )-b^{\prime }(\theta )d^{\prime \prime }(\theta )}{d^{\prime }(\theta )}}$

  $=$$-I$

• For example, for a Poisson distribution

  $$f_X(x\mid \theta )=\frac{{\theta }^x{e}^{-\theta }}{x!},$$

  the natural parameter $d(\theta )$ is $\ln \theta$ and $b(\theta )=-\theta$. $c(x)=x$ since Poisson is in canonical form. Then

  $$U(\theta )=-1+\frac{X}{\theta }\text{ and }I=-\mathrm{E}\left[\frac{-X}{{\theta }^2}\right]=\frac{1}{\theta }$$

  as expected.