conditional distribution of multi-variate normal variable

Theorem.

Let $X$ be a random variable, taking values in ${\mathrm{R}}^n$, normally distributedwith a non-singular covariance matrix $\mathrm{\Sigma }$ and a mean of zero.

Suppose $Y$ is defined by $Y\mathrm{=}{B}^{\mathrm{*}}\mathit{}X$ for some linear transformation $B\mathrm{:}{\mathrm{R}}^k\mathrm{\to }{\mathrm{R}}^n$of maximum rank.(${}^{\mathrm{\ast }}$ to denotes the transpose operator.)

Then the distribution of $X$ conditioned on $Y$ is multi-variate normal,with conditional means and covariances of:

$$𝔼[X\mid Y]=\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y,\mathrm{Var}[X\mid Y]=\mathrm{\Sigma }-\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{(\mathrm{\Sigma }B)}^{\ast }.$$

If $k\mathrm{=}\mathrm{1}$, so that $B$ is simply a vector in ${\mathrm{R}}^n$,these formulas reduce to:

$$𝔼[X\mid Y]=\frac{\mathrm{\Sigma }BY}{\mathrm{Var}[Y]},\mathrm{Var}[X\mid Y]=\mathrm{\Sigma }-\frac{\mathrm{\Sigma }B{B}^{\ast }\mathrm{\Sigma }}{\mathrm{Var}[Y]}.$$

If $X$ does not have zero mean, then theformula for $\mathrm{E}\mathrm{[}X\mathrm{\mid }Y\mathrm{]}$is modified by adding $\mathrm{E}\mathit{}\mathrm{[}X\mathrm{]}$ and replacing $Y$ by $Y\mathrm{-}\mathrm{E}\mathit{}\mathrm{[}Y\mathrm{]}$,and the formula for $\mathrm{Var}\mathit{}\mathrm{[}X\mathrm{\mid }Y\mathrm{]}$ is unchanged.

Proof.

We split up $X$ into two stochastically independent parts,the first part containing exactly the information embodied in $Y$.Then the conditional distribution of $X$ given $Y$ is simplythe unconditional distribution of the second part that is independent of $Y$.

To this end, we firstchange variables to express everything in terms of a standard multi-variate normal$Z$. Let $A:{ℝ}^n\to {ℝ}^n$ be a “square root” factorization ofthe covariance matrix $\mathrm{\Sigma }$,so that:

$$A{A}^{\ast }=\mathrm{\Sigma },Z=A^{-1}X,X=AZ,Y=B^{\ast }AZ.$$

We let $H:{ℝ}^n\to {ℝ}^n$ be the orthogonal projection onto the range of$A^{\ast }B:{ℝ}^k\to {ℝ}^n$, and decompose $Z$ into orthogonal components:

$$Z=HZ+(I-H)Z.$$

It is intuitively obvious that orthogonalityof the two random normal vectors implies their stochastic independence.To show this formally, observe that the Gaussian density function for $Z$factors into a product:

$${(2\pi )}^{-n/2}\exp \left(-\frac{1}{2}{\parallel z\parallel }^2\right)={(2\pi )}^{-n/2}\exp \left(-\frac{1}{2}{\parallel Hz\parallel }^2\right)\exp \left(-\frac{1}{2}{\parallel (I-H)z\parallel }^2\right).$$

We can construct an orthonormal system of coordinates on ${ℝ}^n$under which the components for $Hz$ arecompletely disjoint from those components of $(I-H)z$.On the other hand, the densities for $Z$, $HZ$, and $(I-H)Z$remain invariant even after changing coordinates,because they are radially symmetric.Hence the variables $HZ$ and $(I-H)Z$ are separable in their joint densityand they are independent.

$HZ$ embodies the information in the linear combination $Y=B^{\ast }AZ$.For we have the identity:

$$Y=\left(B^{\ast }A\right)Z=\left(B^{\ast }A\right)\left(HZ+(I-H)Z\right)=\left(B^{\ast }A\right)HZ+0.$$

The last term is null because $(I-H)Z$ is orthogonal to the range of $A^{\ast }B$by definition. (Equivalently, $(I-H)Z$ lies in the kernel of ${(A^{\ast }B)}^{\ast }=B^{\ast }A$.)Thus $Y$ can always be recovered by a linear transformation on$HZ$.

Conversely, $Y$ completely determines $HZ$,from the analytical expression for $H$ that we now give.In general, the orthogonal projection onto the range of an injectivetransformation $T$is $T{(T^{\ast }T)}^{-1}{T}^{\ast }$. Applying this to $T=A^{\ast }B$, we have

$H$$=A^{\ast }B{\left(B^{\ast }A{A}^{\ast }B\right)}^{-1}{B}^{\ast }A$

$=A^{\ast }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{B}^{\ast }A.$

We see that $HZ=A^{\ast }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y$.

We have proved that conditioning on $Y$ and $HZ$ are equivalent, and so:

$$𝔼[Z\mid Y]=𝔼[Z\mid HZ]=𝔼[HZ+(I-H)Z\mid HZ]=HZ+0,$$

and

$\mathrm{Var}[Z\mid Y]=\mathrm{Var}[Z\mid HZ]$$=\mathrm{Var}[HZ+(I-H)Z\mid HZ]$

$=0+\mathrm{Var}[(I-H)Z]$

$=𝔼\left[(I-H)Z{Z}^{\ast }{(I-H)}^{\ast }\right]$

$=(I-H){(I-H)}^{\ast }$

$=I-H-H^{\ast }+H{H}^{\ast }=I-H,$

using the defining property$H^2=H=H^{\ast }$of orthogonal projections.

Now we express the result in terms of $X$, and remove the dependence on the transformation $A$(which is not uniquely defined from the covariance matrix):

$$𝔼[X\mid Y]=A𝔼[Z\mid Y]=AHZ=\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y$$

and

$$\mathrm{Var}[X\mid Y]=A\mathrm{Var}[Z\mid Y]A^{\ast }=A{A}^{\ast }-AH{A}^{\ast }=\mathrm{\Sigma }-\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{B}^{\ast }\mathrm{\Sigma }.$$

Of course, the conditional distribution of $X$ given $Y$ is the sameas that of $(I-H)Z$, which is multi-variate normal.

The formula in the statement of this theorem, for the single-dimensional case,follows from substituting in $\mathrm{Var}[Y]=\mathrm{Var}[B^{\ast }X]=B^{\ast }\mathrm{\Sigma }B$.The formula for when $X$ does not have zeromean follows from applying the base case to the shiftedvariable $X-𝔼[X]$.∎