Principal component analysis (PCA) finds orthogonal directions that summarize the variation in a data set. Instead of describing every observation using the original variables, PCA builds new coordinates as weighted combinations of them. The first coordinate captures as much variance as possible; each later coordinate captures as much of the remaining variance as possible while staying perpendicular to the earlier ones.
The geometric idea of fitting lines and planes to point clouds traces back to Pearson; Hotelling later formalized principal components for statistical variables. [Pearson1901], [Hotelling1933]
Consider two centered sensor measurements that tend to rise together. Their scatter forms a long, tilted cloud: most variation lies along its long axis, with less variation across its width. PCA makes this geometric intuition precise and extends it to any number of variables.
Center the Data First
PCA measures variation around the mean, so subtract the mean of each variable before finding its directions. For observations \(\mathbf{x}_1,\ldots,\mathbf{x}_n\), define their mean and centered values by
\[\bar{\mathbf{x}}=\frac1n\sum_{i=1}^{n}\mathbf{x}_i,\qquad \tilde{\mathbf{x}}_i=\mathbf{x}_i-\bar{\mathbf{x}}.\]
Centering translates the whole cloud without changing its shape or pairwise differences. It places the centroid at the origin, which makes the origin-based projections below measure spread rather than distance from an arbitrary coordinate zero.
Project the Data onto a Direction
Write the \(n\) centered observations of \(d\) variables as the rows of \(\mathbf{X}\in\mathbb{R}^{n\times d}\). A candidate direction is a unit vector \(\mathbf{p}\in\mathbb{R}^d\). Each observation \(\mathbf{x}_i\) projects onto this direction at the scalar coordinate \(z_i=\mathbf{x}_i^T\mathbf{p}\); for all observations together, the scores are \(\mathbf{z}=\mathbf{Xp}\).
The first principal component is the unit direction that makes these scores vary as much as possible. The unit-length constraint matters: without it, multiplying \(\mathbf{p}\) by an arbitrary constant would make the scores and their variance arbitrarily large.
The variance of the projected scores
Because the columns of \(\mathbf{X}\) are centered, the scores also have mean zero. Their sample variance is
\[\operatorname{Var}(\mathbf{z})=\frac{1}{n-1}\mathbf{z}^T\mathbf{z} =\frac{1}{n-1}\mathbf{p}^T\mathbf{X}^T\mathbf{Xp} =\mathbf{p}^T\mathbf{Cp},\qquad \mathbf{C}=\frac{1}{n-1}\mathbf{X}^T\mathbf{X}.\]
The matrix \(\mathbf{C}\) is the sample covariance matrix. Its diagonal entries are the individual variable variances; its off-diagonal entries record how pairs of variables vary together. The PCA question is now precise: which unit vector \(\mathbf{p}\) maximizes the quadratic form \(\mathbf{p}^T\mathbf{Cp}\)?
Why the largest eigenvalue appears
The covariance matrix is symmetric and positive semidefinite, so the Rayleigh quotient says that the maximum of \(\mathbf{p}^T\mathbf{Cp}\) over unit vectors is the largest eigenvalue of \(\mathbf{C}\). Its maximizing direction is the corresponding eigenvector. The same conclusion follows from a constrained optimization: impose \(\mathbf{p}^T\mathbf{p}=1\) with a Lagrange multiplier \(\lambda\), and require a stationary point:
\[\frac{\partial}{\partial\mathbf{p}}\Big[\mathbf{p}^T\mathbf{Cp}-\lambda(\mathbf{p}^T\mathbf{p}-1)\Big] =2\mathbf{Cp}-2\lambda\mathbf{p}=\mathbf{0} \quad\Longleftrightarrow\quad\mathbf{Cp}=\lambda\mathbf{p}.\]
Every stationary direction is therefore an eigenvector, and its captured variance is its eigenvalue. The largest eigenvalue gives the first principal component. The lab below makes this maximization visible by rotating a candidate direction through the same centered point cloud.
Projection lab: rotate the candidate direction
The dashed segments are the errors left after projection. Rotate the teal line: captured variance is largest when the line follows the cloud's long axis.
- Variance captured
- –
- Residual sum of squares
- –
- PC1 direction
- –
The maximizing direction is PC1.
A complete two-variable calculation
Take four centered observations \((2,1),(1,2),(-2,-1),(-1,-2)\). Their data matrix and sample covariance are
\[\mathbf{X}=\begin{bmatrix}2&1\\1&2\\-2&-1\\-1&-2\end{bmatrix},\qquad \mathbf{C}=\frac{1}{3}\mathbf{X}^T\mathbf{X} =\begin{bmatrix}10/3&8/3\\8/3&10/3\end{bmatrix}.\]
The eigenvectors are \(\mathbf{p}_1=(1,1)^T/\sqrt2\) and \(\mathbf{p}_2=(1,-1)^T/\sqrt2\), with eigenvalues \(\lambda_1=6\) and \(\lambda_2=2/3\). Thus PC1 captures \(6/(6+2/3)=90\%\) of the variance. The four score pairs are \((3,1)/\sqrt2,(3,-1)/\sqrt2,(-3,-1)/\sqrt2,(-3,1)/\sqrt2\). Keeping PC1 projects the two positive observations to \((1.5,1.5)\), and the two negative observations to \((-1.5,-1.5)\).
Complete the Orthogonal Basis
After finding \(\mathbf{p}_1\), seek the unit direction \(\mathbf{p}_2\) with the greatest projected variance subject to \(\mathbf{p}_2\perp\mathbf{p}_1\). The same argument applies in that smaller subspace: \(\mathbf{p}_2\) is an eigenvector of \(\mathbf{C}\), specifically the one associated with the second-largest eigenvalue. Repeating the process gives the remaining directions in decreasing order of variance.
Collect the directions as columns of \(\mathbf{P}\). Then \(\mathbf{C}=\mathbf{P}\mathbf{\Lambda}\mathbf{P}^T\), where \(\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_d\ge0\). The transformed observations \(\mathbf{Z}=\mathbf{XP}\) are called scores. Their covariance is diagonal:
\[\operatorname{Cov}(\mathbf{Z})=\mathbf{P}^T\mathbf{CP}=\mathbf{\Lambda}.\]
Scores and loadings answer different questions
A score is an observation's coordinate on a principal axis. A component direction supplies the weights used to form that coordinate. For example, if \(\mathbf{p}_1=(0.96,0.28)^T\), then its score is the linear combination \(z_{i1}=0.96x_{i1}+0.28x_{i2}\). The direction entries are often called loadings; terminology varies, and some conventions scale them by \(\sqrt{\lambda_j}\). The sign of a loading indicates direction, while its magnitude describes the variable's contribution under the chosen scaling.
The components are not perfectly unique. Replacing any \(\mathbf{p}_j\) by \(-\mathbf{p}_j\) changes score signs but leaves variance unchanged. If eigenvalues coincide, any orthonormal basis of their shared eigenspace is valid; the selected axes can rotate within it. These are the same sign and repeated-value ambiguities seen in the singular value decomposition. Neither ambiguity changes the represented subspace or reconstruction quality.
An Equivalent View: Minimizing Reconstruction Error
PCA can also be defined as the best low-dimensional reconstruction. Let the columns of \(\mathbf{W}\in\mathbb{R}^{d\times k}\) be an orthonormal basis for a candidate \(k\)-dimensional subspace. Project an observation into that subspace, then reconstruct it in the original coordinates. Choose \(\mathbf{W}\) to minimize the total squared reconstruction error:
\[\min_{\mathbf{W}^T\mathbf{W}=\mathbf{I}_k}\ \sum_{i=1}^n\big\|\mathbf{x}_i-\mathbf{W}\mathbf{W}^T\mathbf{x}_i\big\|_2^2.\]
The projected coordinates are \(\mathbf{W}^T\mathbf{x}_i\); the reconstructed point is \(\mathbf{WW}^T\mathbf{x}_i\). The residual from the observation to its reconstruction is perpendicular to the chosen subspace. For one observation, this right angle gives the Pythagorean identity:
\[\|\mathbf{x}_i\|_2^2=\big\|\mathbf{W}^T\mathbf{x}_i\big\|_2^2+\big\|\mathbf{x}_i-\mathbf{WW}^T\mathbf{x}_i\big\|_2^2,\]
The reconstruction lies in the subspace and the residual is perpendicular to it, so their squared lengths add to the squared length of the centered observation:
Summing over observations, \(\sum_i\|\mathbf{x}_i\|_2^2\) is fixed. Minimizing the sum of squared residuals is therefore equivalent to maximizing the captured energy \(\sum_i\|\mathbf{W}^T\mathbf{x}_i\|_2^2=\operatorname{trace}(\mathbf{W}^T\mathbf{X}^T\mathbf{XW})\), or the total variance along the chosen directions.
The maximizing subspace is spanned by the top \(k\) eigenvectors of \(\mathbf{C}\). Its minimum reconstruction error is \(\sum_{j=k+1}^d\lambda_j\), the sum of the discarded eigenvalues. Thus variance maximization and least-squares reconstruction select the same PCA subspace.
Choosing the Number of Components: Explained Variance
The covariance matrix's trace is the sum of the original variables' variances. It also equals the sum of its eigenvalues, \(\operatorname{trace}(\mathbf{C})=\sum_{j=1}^d\lambda_j\), so it is the total variance to be distributed across the components.
The fraction assigned to component \(j\) is its eigenvalue divided by that total:
\[\text{explained variance of component }j=\frac{\lambda_j}{\sum_{i=1}^d\lambda_i},\]
This is a share of total variance, not a probability or a general goodness-of-fit score. The shares sum to \(100\%\) across all components. Keeping the first \(k\) retains the cumulative fraction \(\sum_{j=1}^{k}\lambda_j\big/\sum_{j=1}^{d}\lambda_j\).
For example, suppose a five-variable dataset has eigenvalues \(\lambda=(11.81,\,5.19,\,1.67,\,0.47,\,0.05)\). The individual shares are \((61.5\%,\,27.1\%,\,8.7\%,\,2.4\%,\,0.3\%)\): the first two retain \(88.6\%\), while the first three retain \(97.3\%\).
The bars show each component's share; the dotted line traces the cumulative total. Here the 90% threshold is crossed at component three.
A 90% or 99% threshold is only a rule of thumb. It measures retained variance, not usefulness for a particular task. For classification or prediction, select \(k\) using validation performance; low-variance components may contain either noise or information relevant to the task.
Compute PCA with the Singular Value Decomposition
Let the centered data matrix have the singular value decomposition \(\mathbf{X}=\mathbf{U\Sigma V}^T\). Then \(\mathbf{X}^T\mathbf{X}=\mathbf{V\Sigma}^T\mathbf{\Sigma V}^T\). The columns of \(\mathbf{V}\) are therefore the covariance eigenvectors, and the eigenvalues are \(\sigma_j^2/(n-1)\). In other words, the PCA directions are the right singular vectors of the centered data matrix.
In computation, applying SVD directly avoids explicitly forming \(\mathbf{X}^T\mathbf{X}\). Forming this product squares the condition number and can reduce numerical accuracy. Keeping only the top \(k\) singular directions also gives the best rank-\(k\) approximation in the Frobenius norm, by the Eckart–Young theorem [EckartYoung1936]. Thus PCA reconstruction and optimal low-rank matrix approximation are the same result viewed from statistics and linear algebra.
When Variables Outnumber Observations
When \(n\ll d\), the covariance matrix is much larger than necessary: \(\mathbf{C}\) is \(d\times d\), while \(\mathbf{X}\) has at most \(n\) nonzero singular values. A \(100\times100\)-pixel image has \(d=10{,}000\) variables, so its covariance matrix would have \(10^8\) entries. Instead, use the smaller \(n\times n\) Gram matrix \(\mathbf{XX}^T\):
\[\mathbf{XX}^T\mathbf{u}_i=\sigma_i^2\mathbf{u}_i,\]
The eigenvalues are \(\sigma_i^2\), and the corresponding left singular vectors are \(\mathbf{u}_i\). Recover each right singular vector, hence each PCA direction, from \(\mathbf{X}^T\mathbf{u}_i=\sigma_i\mathbf{v}_i\):
\[\boxed{\mathbf{v}_i=\frac{\mathbf{X}^T\mathbf{u}_i}{\sigma_i}}\qquad(\sigma_i>0),\]
This dual formulation costs based on \(n\), not \(d\), and is useful for data such as face images where there are many pixels but relatively few observations. For very large problems where even the smaller matrix is too large, iterative methods can estimate leading components without forming a covariance matrix; the EigenGame chapter explores one such approach.
Online PCA: Learning a Direction from Samples
Covariance eigendecomposition and SVD are batch methods: they process a data matrix or its covariance as a whole. If centered observations arrive one at a time, an online rule can update an estimate after each sample without storing the complete data set. This changes how the component is computed, not what PCA means.
Let \(\mathbf{w}_t\) be a unit-length estimate of PC1 and let \(\mathbf{x}_t\) be the next centered sample. A linear unit produces the scalar response \(y_t=\mathbf{w}_t^T\mathbf{x}_t\). A simple Hebbian update adds \(\eta y_t\mathbf{x}_t\) to the weights: samples that strongly activate the unit pull its direction toward themselves. This uncorrected rule also increases the weight norm without bound.
Oja's rule adds a normalization term proportional to the squared response [Oja1982]:
\[\mathbf{w}_{t+1}=\mathbf{w}_t+\eta y_t\bigl(\mathbf{x}_t-y_t\mathbf{w}_t\bigr),\qquad y_t=\mathbf{w}_t^T\mathbf{x}_t.\]
For samples with covariance \(\mathbf{C}\), the expected change for a small learning rate is
\[\mathbb{E}[\mathbf{w}_{t+1}-\mathbf{w}_t] =\eta\bigl(\mathbf{C}\mathbf{w}_t-(\mathbf{w}_t^T\mathbf{C}\mathbf{w}_t)\mathbf{w}_t\bigr).\]
At a unit-length equilibrium, the expected update vanishes only when \(\mathbf{C}\mathbf{w}=\lambda\mathbf{w}\): the weight direction is an eigenvector. When the largest eigenvalue is distinct and the learning rate follows suitable stochastic-approximation conditions, Oja's dynamics converge toward the leading component, up to its arbitrary sign. The normalization term keeps the vector bounded while the covariance term attracts it toward high-variance directions.
Several components: Sanger's generalized Hebbian rule
For multiple ordered components, Sanger's rule updates each unit while progressively subtracting the contributions already represented by it and the preceding units. With \(y_j=\mathbf{w}_j^T\mathbf{x}_t\), one form is [Sanger1989]:
\[\mathbf{w}_j\leftarrow\mathbf{w}_j+\eta y_j\left(\mathbf{x}_t-\sum_{k=1}^{j}y_k\mathbf{w}_k\right).\]
Under suitable conditions, this generalized Hebbian algorithm learns the leading principal directions in order. It is useful for streaming or distributed settings, but it does not replace the exact batch SVD when that is practical; its convergence depends on the data stream, initialization, and learning-rate schedule.
Whitening: Making Every Direction Equally Important
PCA decorrelates its scores but does not make their variances equal: component \(j\) has variance \(\lambda_j\). Some later methods work better when every retained direction has unit variance. Dividing each score by \(\sqrt{\lambda_j}\) is called whitening or sphering.
Keep only the \(r\) components with positive eigenvalues. With their eigenvectors in \(\mathbf{P}_r\) and eigenvalues in \(\mathbf{\Lambda}_r\), whitened scores are
\[\mathbf{Z}_{\text{white}}=\mathbf{X}\mathbf{P}_r\mathbf{\Lambda}_r^{-1/2},\qquad \operatorname{Cov}(\mathbf{Z}_{\text{white}})=\mathbf{I}_r.\]
Zero-eigenvalue directions cannot be divided by zero, so they are omitted. The result is white in the retained component space. A symmetric transform back in the original coordinates uses \(\mathbf{X}\mathbf{P}_r\mathbf{\Lambda}_r^{-1/2}\mathbf{P}_r^T\); if components were discarded, its covariance is a projection onto the retained subspace rather than the full identity matrix. Whitening is useful when a downstream method needs comparable variance in each input direction, but it deliberately amplifies low-variance directions and can therefore amplify noise.
Principal Components Are Not a Regression Line
The first principal component can resemble a regression line, but they solve different problems. Ordinary least-squares regression of \(y\) on \(x\) minimizes squared vertical errors and treats \(x\) as fixed. PCA minimizes squared perpendicular errors and treats both variables symmetrically.
For the correlated cloud in the first figure, the regression slope is \(0.828\), while the PC1 slope is \(0.839\). They are close here, but diverge as perpendicular noise grows relative to the spread along the main axis.
Assumptions and Limitations
PCA captures linear structure
PCA finds straight directions and subspaces. A data set that lies along a curved path may not be summarized well by a few linear components, even when the path has only one intrinsic degree of freedom. Kernel PCA applies PCA to an implicit feature space defined by a kernel; manifold-learning methods instead focus on preserving local neighborhoods. These methods optimize different notions of structure.
Feature scales and outliers matter
Covariance PCA uses the units supplied with the features. Measuring one variable in millimetres instead of metres multiplies its numerical variance by a million and can rotate PC1 toward that coordinate, even though the measurements describe the same physical data. Standardizing each feature to unit variance replaces covariance PCA with correlation PCA. That is useful when units are arbitrary and comparable importance is intended, but it changes the question when the original scales carry meaning. Outliers also have strong influence because covariance uses squared deviations.
Uncorrelated does not mean independent
The scores are uncorrelated by construction. Unless the joint distribution is Gaussian, zero covariance does not imply statistical independence. Independent component analysis (ICA) targets independence directly; its components need not be orthogonal or ordered by variance.
PCA does not use class labels
PCA only sees the covariance matrix. The direction of greatest overall variance may have little to do with separating known classes. When labels are available and classification is the goal, linear discriminant analysis instead seeks directions with large between-class scatter relative to within-class scatter.
Applications
- Dimensionality reduction and visualization. Projecting onto the first two or three components gives a low-dimensional view that retains as much total variance as any linear projection of the same dimension. For MNIST, each \(28\times28\)-pixel image is a point in \(\mathbb{R}^{784}\); PCA finds directions of pixel variation, not digit classes. A plot of the first two scores may show structure, but different digits can overlap because their labels do not enter the calculation. [LeCun1998]
- Noise reduction and compression. Discarding trailing low-variance components can reduce noise; retaining only leading scores and directions compresses the data. This is the best rank-\(k\) approximation characterized by the SVD and the Eckart–Young theorem [EckartYoung1936]. Related covariance expansions are also called the Hotelling transform and, in continuous settings, the Karhunen–Loève transform. [Hotelling1933], [Karhunen1947], [Loeve1955]
- Eigenfaces. Treating each face image as one long vector of pixel intensities and running PCA produces directions called eigenfaces; a face is approximated by a short score vector. Sirovich and Kirby introduced this representation [SirovichKirby1987], and Turk and Pentland developed it for recognition [TurkPentland1991]. A new face can be represented by its scores and compared with stored face scores, rather than compared pixel by pixel. PCA on corresponding facial landmarks instead models shape variation; flexible face models extended this idea beyond pixel appearance [LanitisTaylorCootes1997].
Summary
PCA first centers the observations, then chooses a unit direction that maximizes projected variance. The covariance matrix turns this optimization into an eigenvalue problem: eigenvectors give component directions, and eigenvalues give their variances. Orthogonal components provide uncorrelated scores; keeping only the leading ones gives the best linear least-squares reconstruction at that dimension.
The SVD computes the same directions without explicitly forming the covariance matrix, while dual PCA can be more efficient when observations are fewer than variables. Explained variance helps describe information retained, but is not a guarantee of usefulness for a particular task. Scaling, outliers, and linearity shape what PCA finds; online methods such as Oja's and Sanger's rules offer sample-by-sample alternatives when a batch decomposition is impractical.
The following sketch illustrates score coordinates for digit images. It is conceptual, not a measured MNIST projection: PCA uses no digit labels, and its leading directions need not separate classes.
Primary References
- [Pearson1901]Pearson, K. (1901). On Lines and Planes of Closest Fit to Systems of Points in Space. Philosophical Magazine, 2(11), 559–572.
- [Hotelling1933]Hotelling, H. (1933). Analysis of a Complex of Statistical Variables into Principal Components, Part I. Journal of Educational Psychology, 24(6), 417–441; Part II, 24(7), 498–520.
- [EckartYoung1936]Eckart, C., & Young, G. (1936). The Approximation of One Matrix by Another of Lower Rank. Psychometrika, 1, 211–218.
- [Oja1982]Oja, E. (1982). Simplified Neuron Model as a Principal Component Analyzer. Journal of Mathematical Biology, 15, 267–273.
- [Sanger1989]Sanger, T. D. (1989). Optimal Unsupervised Learning in a Single-Layer Linear Feedforward Neural Network. Neural Networks, 2(6), 459–473.
- [SirovichKirby1987]Sirovich, L., & Kirby, M. (1987). Low-Dimensional Procedure for the Characterization of Human Faces. Journal of the Optical Society of America A, 4(3), 519–524.
- [TurkPentland1991]Turk, M., & Pentland, A. (1991). Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1), 71–86.
- [LanitisTaylorCootes1997]Lanitis, A., Taylor, C. J., & Cootes, T. F. (1997). Automatic Interpretation and Coding of Face Images Using Flexible Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 743–756.
- [Karhunen1947]Karhunen, K. (1947). Über lineare Methoden in der Wahrscheinlichkeitsrechnung. Annales Academiae Scientiarum Fennicae, Series A I, 37.
- [Loeve1955]Loève, M. (1955). Probability Theory. D. Van Nostrand Company.
- [LeCun1998]LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11), 2278–2324.