chi-squared statistic

Let $X$ be a discrete random variable with $m$ possible outcomes$x_1,\mathrm{\dots },x_m$ with probability of each outcome$\mathrm{P}(X=x_i)=p_i$.

$n$ independent observations are obtained where each observation hasthe same distribution as $X$. Bin the observations into $m$ groups,so that each group contains all observations having the same outcome$x_i$. Next, count the number of observations in each group to get$n_1,\mathrm{\dots },n_k$ corresponding to the outcomes $x_1,\mathrm{\dots },x_k$, sothat $n=\sum n_i$. It is desired to find out how close the actualnumber of outcomes $n_i$ are to their expected values $n{p}_i$.

Intuitively, this “closeness” depends on how big the sample is,and how large the deviations are between the observed and theexpected, for all categories. The value

${\chi }^2={\displaystyle \sum _{i=1}^m}{\displaystyle \frac{{(n_i-n{p}_i)}^2}{n{p}_i}},$(1)

called the ${\chi }^2$ statistic, or the chi-squaredstatistic, is such a measure of “closeness”. It is also known asthe Pearson-chi-squared statistic, in honor of the Englishstatistician Karl Pearson, who showed that (1) has approximately achi-squared distribution (http://planetmath.org/ChiSquaredRandomVariable) with$m-1$ degrees of freedom. The degree of freedom depends on thenumber of free variables in ${\chi }^2$, and is not always $m-1$, as wewill see in Example $3$.

Usually, ${\chi }^2$ statistic is utilized in hypothesis testing, wherethe null hypothesis specifies that the actual equals the expected. Alarge value of ${\chi }^2$ means either the deviations from theexpectations are large or the sample is small, and therefore, eitherthe null hypothesis should be rejected or there is not enoughinformation to give a meaningful interpretation. How large of adeviation, compared to the sample size, is enough to reject the nullhypothesis depends on the degree of freedom of chi-squareddistribution of ${\chi }^2$ and the specified critical values.

Examples.

1. Suppose a coin is tossed 10 times and 7 heads are observed.We would like to know if the coin is fair based on theobservations. We have the following hypothesis:

   $$H_0:p=\frac{1}{2}\mathit{\hspace{1em}\hspace{1em}}{H}_1:p\ne \frac{1}{2}.$$

   Break up the observations into two groups: heads and tails. Then,according to $H_0$,

   $${\chi }^2=\frac{{(7-5)}^2}{5}+\frac{{(3-5)}^2}{5}=1.60.$$

   Checking the table of critical values of chi-squared distributions,we see that at degree of freedom $=1$, there is a 0.100 chance that the${\chi }^2$ value is higher than 2.706. Since , we maynot want to reject the null hypothesis. However, we may not wantto outrightly accept it either simply because the sample size is not verylarge.

2. Now, a coin is tossed 100 times and 70 heads are observed.Using the same null hypothesis as above,

   $${\chi }^2=\frac{{(70-50)}^2}{50}+\frac{{(30-50)}^2}{50}=16.00.$$

   Even at p-value $=0.005$, the corresponding critical value of 7.879is quite a bit smaller than 16. So we will reject the nullhypothesis even at confidence level 99.5%($=1-$p-value).

3. ${\chi }^2$ statistic can be used in non-parametric situations aswell, particularly, in contingency tables. Three dice of varyingsizes are each tossed 100 times and the top faces are recorded. Theresults of the count of each possible value of the top face, foreach die is summarized in the following table:

   Die$\backslash \backslash$top face	1	2	3	4	5	6	all
   Die 1	16	19	17	15	19	14	100
   Die 2	17	18	14	13	22	16	100
   Die 3	12	20	19	18	20	11	100
   All dice	45	57	50	46	61	41	300

   Let $X_i=$ count of top face$=i$, and $Y_j=$ Die $j$. Next, wewant to test the following hypotheses:

   $$H_0:X_i\text{ is independent of }{Y}_j\mathit{\hspace{1em}\hspace{1em}}{H}_1:\text{otherwise}.$$

   Since we do not know the exact distribution ofthe top faces, we approximate the distribution by using the lastrow. For example, the (marginal) probability that top face = 1 is$\frac{45}{300}=0.15$. This says that the probability that top face = 1in Die $i$ = $0.15\times \frac{1}{3}=0.05$. Then, based on thenull hypothesis, we have the following table of “expected count”:

   Die$\backslash \backslash$top face	1	2	3	4	5	6
   Die 1	15.0	19.0	16.7	15.3	20.3	13.7
   Die 2	15.0	19.0	16.7	15.3	20.3	13.7
   Die 3	15.0	19.0	16.7	15.3	20.3	13.7

   For each die, we can compute the ${\chi }^2$. For instance, for thefirst die,

   ${\chi }^2$$=$${\displaystyle \frac{{(16-15.0)}^2}{15.0}}+{\displaystyle \frac{{(19-19.0)}^2}{19.0}}+{\displaystyle \frac{{(17-16.7)}^2}{16.7}}+$

   ${\displaystyle \frac{{(15-15.3)}^2}{15.3}}+{\displaystyle \frac{{(19-20.3)}^2}{20.3}}+{\displaystyle \frac{{(14-13.7)}^2}{13.7}}$

   $=$$0.176$

   The results are summarized in the following

   	${\chi }^2$	degrees of freedom
   Die 1	0.176	5
   Die 2	1.636	5
   Die 3	1.969	0
   All dice	3.781	10

   Note that the degree of freedom for the last dice is 0 because theexpected counts in the last row are completely determined by thosein the first two rows (and the totals). Looking up the table, wesee that there is a $90\%$ that the value of ${\chi }^2$ will begreater than $4.865$, and since , we accept the nullhypothesis: the outcomes of the tosses have no bearing on which dieis tossed.

Remark. In general, for a $p\times q$ 2-way contingencytable, the ${\chi }^2$ statistic is given by

${\chi }^2={\displaystyle \sum _{i=1}^p}{\displaystyle \sum _{j=1}^q}{\displaystyle \frac{{(n_{ij}-m_{ij})}^2}{m_{ij}}},$(2)

where $n_{ij}$ and $m_{ij}$ are the actual and expected counts inCell $(i,j)$. When the sample is large, ${\chi }^2$ has a chi-squareddistribution with $(p-1)(q-1)$ degrees of freedom. In particular,when testing for the independence between two categorical variables,the expected count $m_{ij}$ is

$$m_{ij}=\frac{n_{i*}{n}_{*j}}{n},\text{ where }{n}_{i*}=\sum _{j=1}^q{n}_{ij},n_{*j}=\sum _{i=1}^p{n}_{ij},\text{ and }n=\sum _{i=1}^p\sum _{j=1}^q{n}_{ij}.$$