sufficient statistic

Let $\{f_{\theta }\}$ be a statistical model with parameter$\theta$. Let $𝑿=(X_1,\mathrm{\dots },X_n)$ be a random vectorof random variables representing $n$ observations. A statistic $T=T(𝑿)$ of $𝑿$ for the parameter $\theta$ is called asufficient statistic, or a sufficient estimator, ifthe conditional probability distribution of $𝑿$ given$T(𝑿)=t$ is not a function of $\theta$ (equivalently,does not depend on $\theta$).

In other words, all the information about the unknown parameter$\theta$ is captured in the sufficient statistic $T$. If, say, weare interested in finding out the percentage of defective lightbulbs in a shipment of new ones, it is enough, or sufficient,to count the number of defective ones (sum of the $X_i$’s), ratherthan worrying about which individual light bulbs are the defectiveones (the vector $(X_1,\mathrm{\dots },X_n)$). By taking the sum, a certain“reduction” of data has been achieved.

Examples

1. Let $X_1,\mathrm{\dots },X_n$ be $n$ independent observations from auniform distribution on integers $1,\mathrm{\dots },\theta$. Let$T=\max \{X_1,\mathrm{\dots },X_n\}$ be a statistic for $\theta$.Then the conditional probability distribution of$𝑿=(X_1,\mathrm{\dots },X_n)$ given $T=t$ is

   $$P(𝑿\mid t)=\frac{P(X_1=x_1,\mathrm{\dots },X_n=x_n,\max \{X_n\}=t)}{P(\max \{X_n\}=t)}.$$

   The numerator is $0$ if$\max \{x_n\}\ne t$. So in this case,$P(𝑿\mid t)=0$ and is not a function of $\theta$.Otherwise, the numerator is ${\theta }^{-n}$ and $P(𝑿\mid t)$ becomes

   $$\frac{{\theta }^{-n}}{P(\max \{X_n\}=t)}={({\theta }^{n}P(X_{(1)}\le \mathrm{\cdots }\le X_{(n)}=t))}^{-1},$$

   where$X_{(i)}$’s are the rearrangements of the $X_i$’s in anon-decreasing order from $i=1$ to $n$. For the denominator, we first note that

   $P(X_{(1)}\le \mathrm{\cdots }\le X_{(n)}=t)$$=$

   $=$$P(X_{(1)}\le \mathrm{\cdots }\le X_{(n)}\le t)-P(X_{(1)}\le \mathrm{\cdots }\le X_{(n)}\le t-1).$

   From the above equation, we find that there are $t^n-{(t-1)}^n$ ways to form non-decreasing finite sequences of $n$ positive integers such that the maximum of the sequence is$t$. So

   $${({\theta }^{n}P(X_{(1)}\le \mathrm{\cdots }\le X_{(n)}=t))}^{-1}={({\theta }^n(t^n-{(t-1)}^n){\theta }^{-n})}^{-1}={(t^n-{(t-1)}^n)}^{-1}$$

   again is not a function of $\theta$. Therefore, $T=\max \{X_i\}$ is asufficient statistic for $\theta$.Here, we see that a reduction of data has been achieved by takingonly the largest member of set of observations, not the entire set.

2. If we set $T(X_1,\mathrm{\dots },X_n)=(X_1,\mathrm{\dots },X_n)$, then we seethat $T$ is trivially a sufficient statistic for anyparameter $\theta$. The conditional probability distribution of$(X_1,\mathrm{\dots },X_n)$ given $T$ is 1. Even though this is a sufficientstatistic by definition (of course, the individual observationsprovide as much information there is to know about $\theta$ aspossible), and there is no loss of data in $T$ (which is simply alist of all observations), there is really no reduction of data tospeak of here.

3. The sample mean

   $$\overline{X}=\frac{X_1+\mathrm{\cdots }+X_n}{n}$$

   of $n$ independent observations from a normal distribution$N(\mu ,{\sigma }^2)$ (both $\mu$ and ${\sigma }^2$ unknown) is asufficient statistic for $\mu$. This is the result of thefactorization criterion. Similarly, one sees that any partition ofthe sum of $n$ observations $X_i$ into $m$ subtotals is a sufficientstatistic for $\mu$. For instance,

   $$T(X_1,\mathrm{\dots },X_n)=(\sum _{i=1}^j{X}_i,\sum _{i=j+1}^k{X}_i,\sum _{i=k+1}^n{X}_i)$$

   is a sufficient statistic for $\mu$.

4. Again, assume there are $n$ independent observations $X_i$ froma normal distribution $N(\mu ,{\sigma }^2)$ with unknown mean andvariance. The sample variance

   $$\frac{1}{n-1}\sum _{i=1}^n{(X_i-\overline{X})}^2$$

   is not asufficient statistic for ${\sigma }^2$. However, if $\mu$ is a knownconstant, then

   $$\frac{1}{n-1}\sum _{i=1}^n{(X_i-\mu )}^2$$

   is a sufficient statisticfor ${\sigma }^2$.

A sufficient statistic for a parameter $\theta$ is calleda minimal sufficient statistic if it can be expressed as afunction of any sufficient statistic for $\theta$.

Example. In example $3$ above, both the sample mean$\overline{X}$ and the finite sum $S=X_1+\mathrm{\cdots }+X_n$ are minimalsufficient statistics for the mean $\mu$. Since, by thefactorization criterion, any sufficient statistic $T$ for $\mu$ is avector whose coordinates form a partition of the finite sum, takingthe sum of these coordinates is just the finite sum $S$. So, wehave just expressed $S$ as a function of $T$. Therefore, $S$ isminimal. Similarly, $\overline{X}$ is minimal.

Two sufficient statistics $T_1,T_2$ for a parameter $\theta$ aresaid to be equivalent provided that there is a bijection $g$ suchthat $g\circ T_1=T_2$. $\overline{X}$ and $S$ from the aboveexample are two equivalent sufficient statistics. Two minimal sufficient statistics for the same parameter are equivalent.

Title	sufficient statistic
Canonical name	SufficientStatistic
Date of creation	2013-03-22 15:02:42
Last modified on	2013-03-22 15:02:42
Owner	CWoo (3771)
Last modified by	CWoo (3771)
Numerical id	11
Author	CWoo (3771)
Entry type	Definition
Classification	msc 62B05
Synonym	sufficient estimator
Synonym	minimally sufficient statistic
Synonym	minimal sufficient
Synonym	minimally sufficient
Defines	minimal sufficient statistic
Defines	equivalent statistic