Book contents
Contents
raw Math

Principal component analysis (PCA) finds orthogonal directions that summarize the variation in a data set. Instead of describing every observation using the original variables, PCA builds new coordinates as weighted combinations of them. The first coordinate captures as much variance as possible; each later coordinate captures as much of the remaining variance as possible while staying perpendicular to the earlier ones.

The geometric idea of fitting lines and planes to point clouds traces back to Pearson; Hotelling later formalized principal components for statistical variables. [Pearson1901], [Hotelling1933]

Consider two centered sensor measurements that tend to rise together. Their scatter forms a long, tilted cloud: most variation lies along its long axis, with less variation across its width. PCA makes this geometric intuition precise and extends it to any number of variables.

PC1 PC2 mean
PC1 follows the direction of greatest spread; PC2 is perpendicular. Dashed segments are the residuals left when a point is represented using PC1 alone.

Center the Data First

PCA measures variation around the mean, so subtract the mean of each variable before finding its directions. For observations \(\mathbf{x}_1,\ldots,\mathbf{x}_n\), define their mean and centered values by

\[\bar{\mathbf{x}}=\frac1n\sum_{i=1}^{n}\mathbf{x}_i,\qquad \tilde{\mathbf{x}}_i=\mathbf{x}_i-\bar{\mathbf{x}}.\]

Centering translates the whole cloud without changing its shape or pairwise differences. It places the centroid at the origin, which makes the origin-based projections below measure spread rather than distance from an arbitrary coordinate zero.

Original coordinatesAfter centering meanorigin = mean
Subtracting the mean changes the cloud's location, not its covariance directions. The red crosses mark the mean before and after centering.

Project the Data onto a Direction

Write the \(n\) centered observations of \(d\) variables as the rows of \(\mathbf{X}\in\mathbb{R}^{n\times d}\). A candidate direction is a unit vector \(\mathbf{p}\in\mathbb{R}^d\). Each observation \(\mathbf{x}_i\) projects onto this direction at the scalar coordinate \(z_i=\mathbf{x}_i^T\mathbf{p}\); for all observations together, the scores are \(\mathbf{z}=\mathbf{Xp}\).

The first principal component is the unit direction that makes these scores vary as much as possible. The unit-length constraint matters: without it, multiplying \(\mathbf{p}\) by an arbitrary constant would make the scores and their variance arbitrarily large.

The variance of the projected scores

Because the columns of \(\mathbf{X}\) are centered, the scores also have mean zero. Their sample variance is

\[\operatorname{Var}(\mathbf{z})=\frac{1}{n-1}\mathbf{z}^T\mathbf{z} =\frac{1}{n-1}\mathbf{p}^T\mathbf{X}^T\mathbf{Xp} =\mathbf{p}^T\mathbf{Cp},\qquad \mathbf{C}=\frac{1}{n-1}\mathbf{X}^T\mathbf{X}.\]

The matrix \(\mathbf{C}\) is the sample covariance matrix. Its diagonal entries are the individual variable variances; its off-diagonal entries record how pairs of variables vary together. The PCA question is now precise: which unit vector \(\mathbf{p}\) maximizes the quadratic form \(\mathbf{p}^T\mathbf{Cp}\)?

Why the largest eigenvalue appears

The covariance matrix is symmetric and positive semidefinite, so the Rayleigh quotient says that the maximum of \(\mathbf{p}^T\mathbf{Cp}\) over unit vectors is the largest eigenvalue of \(\mathbf{C}\). Its maximizing direction is the corresponding eigenvector. The same conclusion follows from a constrained optimization: impose \(\mathbf{p}^T\mathbf{p}=1\) with a Lagrange multiplier \(\lambda\), and require a stationary point:

\[\frac{\partial}{\partial\mathbf{p}}\Big[\mathbf{p}^T\mathbf{Cp}-\lambda(\mathbf{p}^T\mathbf{p}-1)\Big] =2\mathbf{Cp}-2\lambda\mathbf{p}=\mathbf{0} \quad\Longleftrightarrow\quad\mathbf{Cp}=\lambda\mathbf{p}.\]

Every stationary direction is therefore an eigenvector, and its captured variance is its eigenvalue. The largest eigenvalue gives the first principal component. The lab below makes this maximization visible by rotating a candidate direction through the same centered point cloud.

Projection lab: rotate the candidate direction

The dashed segments are the errors left after projection. Rotate the teal line: captured variance is largest when the line follows the cloud's long axis.

mean x₁x₂
Variance captured
–
Residual sum of squares
–
PC1 direction
–

The maximizing direction is PC1.

A complete two-variable calculation

Take four centered observations \((2,1),(1,2),(-2,-1),(-1,-2)\). Their data matrix and sample covariance are

\[\mathbf{X}=\begin{bmatrix}2&1\\1&2\\-2&-1\\-1&-2\end{bmatrix},\qquad \mathbf{C}=\frac{1}{3}\mathbf{X}^T\mathbf{X} =\begin{bmatrix}10/3&8/3\\8/3&10/3\end{bmatrix}.\]

The eigenvectors are \(\mathbf{p}_1=(1,1)^T/\sqrt2\) and \(\mathbf{p}_2=(1,-1)^T/\sqrt2\), with eigenvalues \(\lambda_1=6\) and \(\lambda_2=2/3\). Thus PC1 captures \(6/(6+2/3)=90\%\) of the variance. The four score pairs are \((3,1)/\sqrt2,(3,-1)/\sqrt2,(-3,-1)/\sqrt2,(-3,1)/\sqrt2\). Keeping PC1 projects the two positive observations to \((1.5,1.5)\), and the two negative observations to \((-1.5,-1.5)\).

PC1Original points (blue) and PC1 projections (orange)Scoreszᵢ₁ = (xᵢ₁ + xᵢ₂) / √2zᵢ₂ = (xᵢ₁ − xᵢ₂) / √2Captured variancePC1: 90% · PC2: 10%Discarding PC2 leaves the dashed residuals.
The two positive points share a PC1 score because their sums are equal; their different PC2 scores measure the discarded perpendicular variation.

Complete the Orthogonal Basis

After finding \(\mathbf{p}_1\), seek the unit direction \(\mathbf{p}_2\) with the greatest projected variance subject to \(\mathbf{p}_2\perp\mathbf{p}_1\). The same argument applies in that smaller subspace: \(\mathbf{p}_2\) is an eigenvector of \(\mathbf{C}\), specifically the one associated with the second-largest eigenvalue. Repeating the process gives the remaining directions in decreasing order of variance.

Collect the directions as columns of \(\mathbf{P}\). Then \(\mathbf{C}=\mathbf{P}\mathbf{\Lambda}\mathbf{P}^T\), where \(\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_d\ge0\). The transformed observations \(\mathbf{Z}=\mathbf{XP}\) are called scores. Their covariance is diagonal:

\[\operatorname{Cov}(\mathbf{Z})=\mathbf{P}^T\mathbf{CP}=\mathbf{\Lambda}.\]

Scores and loadings answer different questions

A score is an observation's coordinate on a principal axis. A component direction supplies the weights used to form that coordinate. For example, if \(\mathbf{p}_1=(0.96,0.28)^T\), then its score is the linear combination \(z_{i1}=0.96x_{i1}+0.28x_{i2}\). The direction entries are often called loadings; terminology varies, and some conventions scale them by \(\sqrt{\lambda_j}\). The sign of a loading indicates direction, while its magnitude describes the variable's contribution under the chosen scaling.

Original variablesPrincipal-component coordinates xᵢ xᵢ₁xᵢ₂ PC1PC2score zᵢ₁
The same observation gets new coordinates by taking dot products with the orthogonal component directions.

The components are not perfectly unique. Replacing any \(\mathbf{p}_j\) by \(-\mathbf{p}_j\) changes score signs but leaves variance unchanged. If eigenvalues coincide, any orthonormal basis of their shared eigenspace is valid; the selected axes can rotate within it. These are the same sign and repeated-value ambiguities seen in the singular value decomposition. Neither ambiguity changes the represented subspace or reconstruction quality.

An Equivalent View: Minimizing Reconstruction Error

PCA can also be defined as the best low-dimensional reconstruction. Let the columns of \(\mathbf{W}\in\mathbb{R}^{d\times k}\) be an orthonormal basis for a candidate \(k\)-dimensional subspace. Project an observation into that subspace, then reconstruct it in the original coordinates. Choose \(\mathbf{W}\) to minimize the total squared reconstruction error:

\[\min_{\mathbf{W}^T\mathbf{W}=\mathbf{I}_k}\ \sum_{i=1}^n\big\|\mathbf{x}_i-\mathbf{W}\mathbf{W}^T\mathbf{x}_i\big\|_2^2.\]

The projected coordinates are \(\mathbf{W}^T\mathbf{x}_i\); the reconstructed point is \(\mathbf{WW}^T\mathbf{x}_i\). The residual from the observation to its reconstruction is perpendicular to the chosen subspace. For one observation, this right angle gives the Pythagorean identity:

\[\|\mathbf{x}_i\|_2^2=\big\|\mathbf{W}^T\mathbf{x}_i\big\|_2^2+\big\|\mathbf{x}_i-\mathbf{WW}^T\mathbf{x}_i\big\|_2^2,\]

The reconstruction lies in the subspace and the residual is perpendicular to it, so their squared lengths add to the squared length of the centered observation:

centered observationprojection onto the subspaceresidualorigin
The projected vector and its perpendicular residual form the legs of a right triangle.

Summing over observations, \(\sum_i\|\mathbf{x}_i\|_2^2\) is fixed. Minimizing the sum of squared residuals is therefore equivalent to maximizing the captured energy \(\sum_i\|\mathbf{W}^T\mathbf{x}_i\|_2^2=\operatorname{trace}(\mathbf{W}^T\mathbf{X}^T\mathbf{XW})\), or the total variance along the chosen directions.

The maximizing subspace is spanned by the top \(k\) eigenvectors of \(\mathbf{C}\). Its minimum reconstruction error is \(\sum_{j=k+1}^d\lambda_j\), the sum of the discarded eigenvalues. Thus variance maximization and least-squares reconstruction select the same PCA subspace.

Choosing the Number of Components: Explained Variance

The covariance matrix's trace is the sum of the original variables' variances. It also equals the sum of its eigenvalues, \(\operatorname{trace}(\mathbf{C})=\sum_{j=1}^d\lambda_j\), so it is the total variance to be distributed across the components.

The fraction assigned to component \(j\) is its eigenvalue divided by that total:

\[\text{explained variance of component }j=\frac{\lambda_j}{\sum_{i=1}^d\lambda_i},\]

This is a share of total variance, not a probability or a general goodness-of-fit score. The shares sum to \(100\%\) across all components. Keeping the first \(k\) retains the cumulative fraction \(\sum_{j=1}^{k}\lambda_j\big/\sum_{j=1}^{d}\lambda_j\).

For example, suppose a five-variable dataset has eigenvalues \(\lambda=(11.81,\,5.19,\,1.67,\,0.47,\,0.05)\). The individual shares are \((61.5\%,\,27.1\%,\,8.7\%,\,2.4\%,\,0.3\%)\): the first two retain \(88.6\%\), while the first three retain \(97.3\%\).

The bars show each component's share; the dotted line traces the cumulative total. Here the 90% threshold is crossed at component three.

90% 61.5% 27.1% 8.7% 2.4% 0.3% PC1 PC2 PC3 PC4 PC5

A 90% or 99% threshold is only a rule of thumb. It measures retained variance, not usefulness for a particular task. For classification or prediction, select \(k\) using validation performance; low-variance components may contain either noise or information relevant to the task.

Compute PCA with the Singular Value Decomposition

Let the centered data matrix have the singular value decomposition \(\mathbf{X}=\mathbf{U\Sigma V}^T\). Then \(\mathbf{X}^T\mathbf{X}=\mathbf{V\Sigma}^T\mathbf{\Sigma V}^T\). The columns of \(\mathbf{V}\) are therefore the covariance eigenvectors, and the eigenvalues are \(\sigma_j^2/(n-1)\). In other words, the PCA directions are the right singular vectors of the centered data matrix.

In computation, applying SVD directly avoids explicitly forming \(\mathbf{X}^T\mathbf{X}\). Forming this product squares the condition number and can reduce numerical accuracy. Keeping only the top \(k\) singular directions also gives the best rank-\(k\) approximation in the Frobenius norm, by the Eckart–Young theorem [EckartYoung1936]. Thus PCA reconstruction and optimal low-rank matrix approximation are the same result viewed from statistics and linear algebra.

When Variables Outnumber Observations

When \(n\ll d\), the covariance matrix is much larger than necessary: \(\mathbf{C}\) is \(d\times d\), while \(\mathbf{X}\) has at most \(n\) nonzero singular values. A \(100\times100\)-pixel image has \(d=10{,}000\) variables, so its covariance matrix would have \(10^8\) entries. Instead, use the smaller \(n\times n\) Gram matrix \(\mathbf{XX}^T\):

\[\mathbf{XX}^T\mathbf{u}_i=\sigma_i^2\mathbf{u}_i,\]

The eigenvalues are \(\sigma_i^2\), and the corresponding left singular vectors are \(\mathbf{u}_i\). Recover each right singular vector, hence each PCA direction, from \(\mathbf{X}^T\mathbf{u}_i=\sigma_i\mathbf{v}_i\):

\[\boxed{\mathbf{v}_i=\frac{\mathbf{X}^T\mathbf{u}_i}{\sigma_i}}\qquad(\sigma_i>0),\]

This dual formulation costs based on \(n\), not \(d\), and is useful for data such as face images where there are many pixels but relatively few observations. For very large problems where even the smaller matrix is too large, iterative methods can estimate leading components without forming a covariance matrix; the EigenGame chapter explores one such approach.

Online PCA: Learning a Direction from Samples

Covariance eigendecomposition and SVD are batch methods: they process a data matrix or its covariance as a whole. If centered observations arrive one at a time, an online rule can update an estimate after each sample without storing the complete data set. This changes how the component is computed, not what PCA means.

Let \(\mathbf{w}_t\) be a unit-length estimate of PC1 and let \(\mathbf{x}_t\) be the next centered sample. A linear unit produces the scalar response \(y_t=\mathbf{w}_t^T\mathbf{x}_t\). A simple Hebbian update adds \(\eta y_t\mathbf{x}_t\) to the weights: samples that strongly activate the unit pull its direction toward themselves. This uncorrected rule also increases the weight norm without bound.

Oja's rule adds a normalization term proportional to the squared response [Oja1982]:

\[\mathbf{w}_{t+1}=\mathbf{w}_t+\eta y_t\bigl(\mathbf{x}_t-y_t\mathbf{w}_t\bigr),\qquad y_t=\mathbf{w}_t^T\mathbf{x}_t.\]

For samples with covariance \(\mathbf{C}\), the expected change for a small learning rate is

\[\mathbb{E}[\mathbf{w}_{t+1}-\mathbf{w}_t] =\eta\bigl(\mathbf{C}\mathbf{w}_t-(\mathbf{w}_t^T\mathbf{C}\mathbf{w}_t)\mathbf{w}_t\bigr).\]

At a unit-length equilibrium, the expected update vanishes only when \(\mathbf{C}\mathbf{w}=\lambda\mathbf{w}\): the weight direction is an eigenvector. When the largest eigenvalue is distinct and the learning rate follows suitable stochastic-approximation conditions, Oja's dynamics converge toward the leading component, up to its arbitrary sign. The normalization term keeps the vector bounded while the covariance term attracts it toward high-variance directions.

PC1 directioncurrent w Each sample nudges w sample xₜupdated w Oja's correction controls its length
The update combines sample-driven attraction with a term that prevents the weight norm from growing indefinitely.

Several components: Sanger's generalized Hebbian rule

For multiple ordered components, Sanger's rule updates each unit while progressively subtracting the contributions already represented by it and the preceding units. With \(y_j=\mathbf{w}_j^T\mathbf{x}_t\), one form is [Sanger1989]:

\[\mathbf{w}_j\leftarrow\mathbf{w}_j+\eta y_j\left(\mathbf{x}_t-\sum_{k=1}^{j}y_k\mathbf{w}_k\right).\]

Under suitable conditions, this generalized Hebbian algorithm learns the leading principal directions in order. It is useful for streaming or distributed settings, but it does not replace the exact batch SVD when that is practical; its convergence depends on the data stream, initialization, and learning-rate schedule.

Whitening: Making Every Direction Equally Important

PCA decorrelates its scores but does not make their variances equal: component \(j\) has variance \(\lambda_j\). Some later methods work better when every retained direction has unit variance. Dividing each score by \(\sqrt{\lambda_j}\) is called whitening or sphering.

Keep only the \(r\) components with positive eigenvalues. With their eigenvectors in \(\mathbf{P}_r\) and eigenvalues in \(\mathbf{\Lambda}_r\), whitened scores are

\[\mathbf{Z}_{\text{white}}=\mathbf{X}\mathbf{P}_r\mathbf{\Lambda}_r^{-1/2},\qquad \operatorname{Cov}(\mathbf{Z}_{\text{white}})=\mathbf{I}_r.\]

Zero-eigenvalue directions cannot be divided by zero, so they are omitted. The result is white in the retained component space. A symmetric transform back in the original coordinates uses \(\mathbf{X}\mathbf{P}_r\mathbf{\Lambda}_r^{-1/2}\mathbf{P}_r^T\); if components were discarded, its covariance is a projection onto the retained subspace rather than the full identity matrix. Whitening is useful when a downstream method needs comparable variance in each input direction, but it deliberately amplifies low-variance directions and can therefore amplify noise.

Principal Components Are Not a Regression Line

The first principal component can resemble a regression line, but they solve different problems. Ordinary least-squares regression of \(y\) on \(x\) minimizes squared vertical errors and treats \(x\) as fixed. PCA minimizes squared perpendicular errors and treats both variables symmetrically.

For the correlated cloud in the first figure, the regression slope is \(0.828\), while the PC1 slope is \(0.839\). They are close here, but diverge as perpendicular noise grows relative to the spread along the main axis.

PCA OLS vertical perpendicular

Assumptions and Limitations

PCA captures linear structure

PCA finds straight directions and subspaces. A data set that lies along a curved path may not be summarized well by a few linear components, even when the path has only one intrinsic degree of freedom. Kernel PCA applies PCA to an implicit feature space defined by a kernel; manifold-learning methods instead focus on preserving local neighborhoods. These methods optimize different notions of structure.

Feature scales and outliers matter

Covariance PCA uses the units supplied with the features. Measuring one variable in millimetres instead of metres multiplies its numerical variance by a million and can rotate PC1 toward that coordinate, even though the measurements describe the same physical data. Standardizing each feature to unit variance replaces covariance PCA with correlation PCA. That is useful when units are arbitrary and comparable importance is intended, but it changes the question when the original scales carry meaning. Outliers also have strong influence because covariance uses squared deviations.

Comparable feature scalesFeature 1 multiplied by 5 PC1 follows both featuresPC1 is pulled toward the rescaled feature
Changing units changes covariance PCA unless features are standardized. Standardization is a modeling choice, not a neutral cosmetic step.

Uncorrelated does not mean independent

The scores are uncorrelated by construction. Unless the joint distribution is Gaussian, zero covariance does not imply statistical independence. Independent component analysis (ICA) targets independence directly; its components need not be orthogonal or ordered by variance.

PCA does not use class labels

PCA only sees the covariance matrix. The direction of greatest overall variance may have little to do with separating known classes. When labels are available and classification is the goal, linear discriminant analysis instead seeks directions with large between-class scatter relative to within-class scatter.

Applications

Summary

PCA first centers the observations, then chooses a unit direction that maximizes projected variance. The covariance matrix turns this optimization into an eigenvalue problem: eigenvectors give component directions, and eigenvalues give their variances. Orthogonal components provide uncorrelated scores; keeping only the leading ones gives the best linear least-squares reconstruction at that dimension.

The SVD computes the same directions without explicitly forming the covariance matrix, while dual PCA can be more efficient when observations are fewer than variables. Explained variance helps describe information retained, but is not a guarantee of usefulness for a particular task. Scaling, outliers, and linearity shape what PCA finds; online methods such as Oja's and Sanger's rules offer sample-by-sample alternatives when a batch decomposition is impractical.

The following sketch illustrates score coordinates for digit images. It is conceptual, not a measured MNIST projection: PCA uses no digit labels, and its leading directions need not separate classes.

PC1 PC2 0 0 0 0 0 0 1 1 1 1 1 1 6 6 6 6 6 6 3 3 3 3 3 3 8 8 8 8 8 8 9 9 9 9 9 9
Conceptual score plot only. Labels can be used to inspect a PCA projection after fitting, but they do not determine the components.

Primary References