likelihood function

Let X=($X_1,\mathrm{\dots },X_n$) be a random vector and

$$\{f_{𝐗}(𝒙\mid 𝜽):𝜽\in \mathrm{\Theta }\}$$

a statistical model parametrized by $𝜽=({\theta }_1,\mathrm{\dots },{\theta }_k)$, the parameter vector in the parameter space $\mathrm{\Theta }$. The likelihood function is a map $L:\mathrm{\Theta }\to ℝ$ given by

$$L(𝜽\mid 𝒙)=f_{𝐗}(𝒙\mid 𝜽).$$

In other words, the likelikhood function is functionally the same in form as a probability density function. However, the emphasis is changed from the $𝒙$ to the $𝜽$. The pdf is a function of the $x$’s while holding the parameters $\theta$’s constant, $L$ is a function of the parameters $\theta$’s, while holding the $x$’s constant.

When there is no confusion, $L(𝜽\mid 𝒙)$ is abbreviated to be $L(𝜽)$.

The parameter vector $\widehat{𝜽}$ such that $L(\widehat{𝜽})\ge L(𝜽)$ for all $𝜽\in \mathrm{\Theta }$ is called a maximum likelihood estimate, or MLE, of $𝜽$.

Many of the density functions are exponential in nature, it is therefore easier to compute the MLE of a likelihoodfunction $L$ by finding the maximum of the natural log of $L$, known as the log-likelihood function:

$$\mathrm{\ell }(𝜽\mid 𝒙)=\ln (L(𝜽\mid 𝒙))$$

due to the monotonicity of the log function.

Examples:

1. A coin is tossed $n$ times and $m$ heads are observed. Assume that the probability of a head after one toss is $\pi$. What is the MLE of $\pi$?

   Solution: Define the outcome of a toss be 0 if a tail is observed and 1 if a head is observed. Next, let $X_i$ be the outcome ofthe $i$th toss. For any single toss, the density function is ${\pi }^x{(1-\pi )}^{1-x}$ where $x\in \{0,1\}$. Assume that the tosses are independent events, then the joint probability density is

   $$f_{𝐗}(𝒙\mid \pi )=\left(\genfrac{}{}{0pt}{}{n}{\mathrm{\Sigma }{x}_i}\right){\pi }^{\mathrm{\Sigma }{x}_i}{(1-\pi )}^{\mathrm{\Sigma }(1-x_i)}=\left(\genfrac{}{}{0pt}{}{n}{m}\right){\pi }^m{(1-\pi )}^{n-m},$$

   which is also the likelihood function $L(\pi )$. Therefore, the log-likelihood function has the form

   $$\mathrm{\ell }(\pi \mid 𝒙)=\mathrm{\ell }(\pi )=\ln \left(\genfrac{}{}{0pt}{}{n}{m}\right)+m\ln (\pi )+(n-m)\ln (1-\pi ).$$

   Using standard calculus, we get that the MLE of $\pi$ is

   $$\widehat{\pi }=\frac{m}{n}=\overline{x}.$$

2. Suppose a sample of $n$ data points $X_i$ are collected. Assume that the $X_i\sim N(\mu ,{\sigma }^2)$ and the $X_i$’s are independent of each other. What is the MLE of the parameter vector $𝜽=(\mu ,{\sigma }^2)$?

   Solution: The joint pdf of the $X_i$, and hence the likelihood function, is

   $$L(𝜽\mid 𝒙)=\frac{1}{{\sigma }^n{(2\pi )}^{n/2}}\exp (-\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^2}).$$

   The log-likelihood function is

   $$\mathrm{\ell }(𝜽\mid 𝒙)=-\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^2}-\frac{n}{2}\ln ({\sigma }^2)-\frac{n}{2}\ln (2\pi ).$$

   Taking the first derivative (gradient), we get

   $$\frac{\partial \mathrm{\ell }}{\partial 𝜽}=(\frac{\mathrm{\Sigma }(x_i-\mu )}{{\sigma }^2},\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^4}-\frac{n}{2{\sigma }^2}).$$

   Setting

   $$\frac{\partial \mathrm{\ell }}{\partial 𝜽}=\mathrm{𝟎}\text{ See score function}$$

   and solve for $𝜽=(\mu ,{\sigma }^2)$ we have

   $$\widehat{𝜽}=(\widehat{\mu },{\widehat{\sigma }}^2)=(\overline{x},\frac{n-1}{n}{s}^2),$$

   where $\overline{x}=\mathrm{\Sigma }{x}_i/n$ is the sample mean and $s^2=\mathrm{\Sigma }{(x_i-\overline{x})}^2/(n-1)$ is the sample variance. Finally, we verify that $\widehat{𝜽}$ is indeed the MLE of $𝜽$ by checking the negativity of the 2nd derivatives (for each parameter).