canonical correlation

Let $X$ be the $(T,n)$ matrix corresponding to the $n$ signals and $Y$ be a $(T,p)$ matrix corresponding to one set of $p$ signals. Time indexes each row of the matrix ($T$ time samples). Let ${\mathrm{\Sigma }}_{11}$ and ${\mathrm{\Sigma }}_{22}$ be the sample covariance matrices of $X$ and $Y$, respectively, and let ${\mathrm{\Sigma }}_{12}={\mathrm{\Sigma }}_{21}^{\prime }$ be the sample covariance matrix between $X$ and $Y$. For simplicity, we suppose that all signals have zero mean.

Canonical correlation analysis (CCA) finds the linear combinations of the column of $X$ and $Y$ that has the largest correlation; i.e., it finds the weight vectors (loadings) $a$ and $b$ that maximize:

$$\rho =\frac{a^{\prime }{\mathrm{\Sigma }}_{12}b}{\sqrt{a^{\prime }{\mathrm{\Sigma }}_{11}a}\sqrt{b^{\prime }{\mathrm{\Sigma }}_{22}b}}.$$(1)

We follow the derivations of Johnson and we do a change of basis: $c={\mathrm{\Sigma }}_{11}^{1/2}a$ and $d={\mathrm{\Sigma }}_{22}^{1/2}b$.

$$\rho =\frac{c^{\prime }{\mathrm{\Sigma }}_{11}^{-1/2}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1/2}d}{\sqrt{c^{\prime }c}\sqrt{d^{\prime }d}}$$(2)

By the Cauchy-Schwartz inequality:

$$\rho \le \frac{\sqrt{c^{\prime }{\mathrm{\Sigma }}_{11}^{-1/2}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1/2}{\mathrm{\Sigma }}_{22}^{-1/2}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1/2}c}\sqrt{d^{\prime }d}}{\sqrt{c^{\prime }c}\sqrt{d^{\prime }d}}=\sqrt{\frac{c^{\prime }{\mathrm{\Sigma }}_{11}^{-1/2}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1/2}c}{c^{\prime }c}}.$$(3)

The inequality above is an equality when ${\mathrm{\Sigma }}_{22}^{-1/2}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1/2}c$ and $d$ are collinear. The right hand side of the expression above is a Rayleigh quotient and it is maximum when $c$ is the eigenvector corresponding to the largest eingenvalue of ${\mathrm{\Sigma }}_{11}^{-1/2}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1/2}$ (we obtain the other rows by using the other eigenvalues in decreasing magnitude). This results if the basis of the CCA. We can compute the two canonical variables: $U_1=Xa$ and $V_1=Yb$.

We can continue this way to find the subsequent vectors