Book contents
Contents
raw Math
RAW Book Machine Learning Gaussian Processes

Gaussian Process Regression

Robert Eisele

A handful of measurements can support countless curves. Gaussian process regression turns that ambiguity into a probability distribution over functions, showing both what the data suggest and how uncertain each prediction remains.

This chapter derives the method from Bayesian conditioning and covariance, then builds up to kernels, posterior prediction, and model selection. Interactive labs let you see how assumptions about smoothness and noise change a fit, and how the same framework assigns uncertainty to an integral.

Many curves fit the same observations

Suppose an unknown function has been measured at three locations. A straight line may pass through those points, but so may many other curves. Between the measurements, and beyond their range, infinitely many trajectories remain possible. The data alone do not determine a unique answer.

A conventional regression method usually returns one preferred curve. A GP instead describes a probability distribution over functions. At each input it gives a mean and uncertainty, while also describing how predictions at different inputs relate to one another.

Many functions fit three observationsFour curves pass through the same three points but differ between observations and near the boundaries. Input xSame data, different plausible curves
We need a model to interpolate between observations. A Gaussian process makes the modeling assumption and the uncertainty it implies explicit.
The central questionWhich trajectories are plausible before seeing the data, and how should observations change that belief?

Probability describes uncertainty

A random variable need not represent physical randomness. A fixed but unknown quantity can also be described by a probability distribution. For continuous variables, a density is not a point probability: the probability of an interval is the area under the density.

\[P(a\le Y\le b)=\int_a^b p(y)\,dy,\qquad \int_{-\infty}^{\infty}p(y)\,dy=1.\]

The mean summarizes a distribution's location; its variance describes its spread. When evidence supports a hypothesis, Bayes' rule updates its distribution:

\[p(f\mid y)=\frac{p(y\mid f)p(f)}{p(y)}.\]

Here, \(p(f)\) is the prior over functions, \(p(y\mid f)\) the observation model, and \(p(f\mid y)\) the posterior. The denominator normalizes the result. This view keeps the observations separate from the assumptions used to interpret them.

The normal distribution as a building block

A normal distribution is determined by its mean \(\mu\) and standard deviation \(\sigma\). It is useful for GP regression because linear combinations of jointly normal variables remain normal, and their conditional distributions have closed-form expressions.

\[p(y\mid\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(y-\mu)^2}{2\sigma^2}\right),\qquad Y\sim\mathcal N(\mu,\sigma^2).\]
Normal distributions with equal means and different uncertaintyA narrow and a broad Gaussian curve share a mean. The central one-standard-deviation interval contains about 68 percent of the mass. μBroad: large σNarrow: small σ
For a normal variable, about 68% of the probability lies in \(\mu\pm\sigma\). A GP prediction uses such a distribution at every input.

For a vector of function values, knowing each variance is not enough; we also need the covariance between locations. O'Hagan gave an early Bayesian treatment of curve fitting and prediction using a prior distribution over functions [OHagan1978].

Covariance connects function values

Covariance \(\operatorname{Cov}(A,B)\) measures whether two quantities tend to lie above or below their means together. Positive values indicate aligned deviations; negative values indicate opposing ones. Zero covariance implies independence only when the variables are jointly Gaussian.

\[\operatorname{Cov}(A,B)=\mathbb E[(A-\mathbb E A)(B-\mathbb E B)].\]

The covariances of function values at \(x_1,\ldots,x_n\) form a matrix \(K\), with \(K_{ij}=\operatorname{Cov}(f(x_i),f(x_j))\). A valid covariance matrix is symmetric and positive semidefinite: for every vector \(a\), \(a^\mathsf TKa\ge0\), because this expression is the variance of the linear combination \(a^\mathsf Tf\).

Covariance matrix of a smooth processA heat map shows high covariance along the diagonal, decreasing with distance.Kᵢⱼ = k(xᵢ,xⱼ)Nearby inputsstrongly coupled valuesDistant inputsweaker association
The diagonal contains variances. The matrix must be valid as a whole; plausible pairwise similarities are not enough.

Observing means conditioning

For two standardized, jointly normal variables \(A\) and \(B\) with correlation \(\rho\):

\[(A,B)^\mathsf T\sim\mathcal N\!\left(\begin{bmatrix}0\\0\end{bmatrix},\begin{bmatrix}1&\rho\\\rho&1\end{bmatrix}\right),\qquad A\mid B=b\sim\mathcal N(\rho b,1-\rho^2).\]

The observation shifts the mean of \(A\) according to \(b\) and \(\rho\), and reduces its variance. If \(\rho=0\), then \(B\) tells us nothing about \(A\). As \(|\rho|\) approaches one, \(A\) becomes nearly determined. For a jointly Gaussian vector split into unobserved values \(u\) and observed values \(v\), the same rule is:

\[\begin{aligned}\mathbb E[u\mid v]&=\mu_u+\Sigma_{uv}\Sigma_{vv}^{-1}(v-\mu_v),\\\operatorname{Cov}(u\mid v)&=\Sigma_{uu}-\Sigma_{uv}\Sigma_{vv}^{-1}\Sigma_{vu}.\end{aligned}\]
An observation updates nearby function valuesA measurement pulls nearby predictions toward its value and reduces uncertainty there.ObservationThe kernel determines how far information travels
Observations do not influence every input equally: the kernel determines how information spreads across the input space.

Bayesian linear regression provides the bridge

A line can be written as a weighted sum of features, for example \(f(x)=w_0+w_1x\). Instead of treating its weights as unknown but fixed numbers, place a Gaussian distribution over them. Each possible set of weights then defines a possible line.

\[w\sim\mathcal N(m_0,\Sigma_0),\qquad y_i=\phi(x_i)^\mathsf Tw+\epsilon_i,\qquad \epsilon_i\sim\mathcal N(0,\sigma_n^2).\]

After observing data, the weight posterior remains Gaussian. With design matrix \(\Phi\):

\[\begin{aligned}S_N&=(\Sigma_0^{-1}+\sigma_n^{-2}\Phi^\mathsf T\Phi)^{-1},\\m_N&=S_N(\Sigma_0^{-1}m_0+\sigma_n^{-2}\Phi^\mathsf Ty),\\w\mid X,y&\sim\mathcal N(m_N,S_N).\end{aligned}\]

A new function value is a linear projection of this Gaussian vector, so it is Gaussian too [WilliamsRasmussen1995]:

\[f(x_*)\mid X,y\sim\mathcal N\!\left(\phi(x_*)^\mathsf Tm_N,\,\phi(x_*)^\mathsf TS_N\phi(x_*)\right).\]

Next, let the number of features grow very large, even infinite, without explicitly storing their weights.

The kernel hides the feature space

The covariance between two function values in a linear weight model is the inner product of their features, weighted by the prior covariance of the weights:

\[k(x,x')=\operatorname{Cov}(f(x),f(x'))=\phi(x)^\mathsf T\Sigma_0\phi(x').\]

If all computations depend only on these inner products, we no longer need to calculate the features individually. Any function \(k(x,x')\) that produces a positive semidefinite covariance matrix for every finite set of inputs can be used directly as a kernel.

From feature space to a kernel evaluationInputs can be mapped to features and compared there, or compared directly through a kernel function.x, x′inputsφ(x), φ(x′)featuresφ(x)ᵀφ(x′)inner productk(x,x′)kernelAlternatively, compute k directly without constructing φ.
A kernel is not just any similarity score. It must define a mathematically valid covariance for arbitrary finite point sets.

The squared-exponential kernel (SE, also called the RBF kernel) in one dimension is

\[k_{\mathrm{SE}}(x,x')=\sigma_f^2\exp\!\left(-\frac{(x-x')^2}{2\ell^2}\right).\]

The amplitude \(\sigma_f\) controls typical function magnitude; the length scale \(\ell\) controls how quickly correlation decays with distance. A large \(\ell\) favors slowly varying functions. The SE kernel is extremely smooth, which is a strong modeling assumption.

KernelCovariance at distance rTypical assumption
SE / RBF\(\sigma_f^2e^{-r^2/(2\ell^2)}\)Very smooth functions
Matérn 3/2\(\sigma_f^2(1+\sqrt3r/\ell)e^{-\sqrt3r/\ell}\)Rougher, once-differentiable paths
Periodic\(\sigma_f^2e^{-2\sin^2(\pi r/p)/\ell^2}\)Repeating patterns with period p

A Gaussian process is a distribution over functions

A function has infinitely many possible values. A Gaussian process avoids storing an infinite table: for every finite selection of inputs, the corresponding function values are jointly Gaussian, and those distributions must be mutually consistent.

\[f\sim\mathcal{GP}(m,k)\quad\Longleftrightarrow\quad (f(x_1),\ldots,f(x_n))^\mathsf T\sim\mathcal N(m(X),K(X,X)).\]

The mean function \(m(x)\) sets the average trajectory; the kernel \(k(x,x')\) sets variances and dependencies. A sampled curve is one realization from the prior, not a “mean plus a little noise.”

Samples from a Gaussian process priorSeveral smooth curves are drawn around a zero mean. The kernel determines how quickly they fluctuate.Each curve is a possible functionPrior mean: 0
Before observing data, many functions may be plausible. The kernel determines which kinds of variation the prior favors.

Deriving the GP regression formula

We observe \(y_i=f(x_i)+\epsilon_i\), with independent Gaussian noise \(\epsilon_i\sim\mathcal N(0,\sigma_n^2)\). The noise variance is added only to the diagonal of the covariance matrix:

\[A=K(X,X)+\sigma_n^2I.\]

A new latent function value \(f_*=f(x_*)\) is jointly Gaussian with the observations. Applying the conditioning rule gives:

\[\begin{aligned}\mu_*(x_*)&=m(x_*)+k(x_*,X)A^{-1}[y-m(X)],\\\sigma_*^2(x_*)&=k(x_*,x_*)-k(x_*,X)A^{-1}k(X,x_*),\\f(x_*)\mid X,y&\sim\mathcal N(\mu_*(x_*),\sigma_*^2(x_*)).\end{aligned}\]

The first term shifts the prediction according to the observation residuals. The second reduces uncertainty to the extent that the new input is correlated with the training points. For a new noisy measurement, add \(\sigma_n^2\) to the latent posterior variance. Uncertainty about \(f(x_*)\) and about a new observation are not the same.

GP regression after several observationsA posterior mean follows noisy data, while an uncertainty band shrinks around training inputs.Posterior mean and pointwise uncertainty
Uncertainty shrinks at training inputs. Between them it depends on the kernel. Far from all data, a stationary model returns toward its prior.
An important implementation detailThe equations contain \(A^{-1}\), but a numerically stable implementation does not form this inverse explicitly. It solves linear systems using a Cholesky factorization.

Interactive GP lab

Change the kernel, length scale, signal amplitude, noise level, and number of observations. The synthetic example has a known underlying function; the GP model sees only the selected measurements.

Regression and covariance

The lighter region is a pointwise 95% interval, not a simultaneous band for the entire curve.
synthetic functionobservationsposterior meanprior mean
Log marginal likelihood—
Posterior variance at center—
Cholesky diagonal—
The left plot shows covariance decaying with distance; the heat map on the right shows covariances between training points.
Log marginal likelihood balances data fit against model complexity. This curve is a one-dimensional slice with the other parameters fixed.

Hyperparameters are uncertain too

The kernel contains parameters such as length scale and amplitude. They can be set using domain knowledge, selected by cross-validation, or optimized using the marginal likelihood. With Gaussian noise, its logarithm is:

\[\log p(y\mid X,\theta)=-\frac12r^\mathsf TA^{-1}r-\frac12\log|A|-\frac n2\log(2\pi),\qquad r=y-m(X).\]

The first term rewards data fit; the determinant term penalizes, roughly speaking, models that spread their probability mass over too many trajectories. MacKay developed this Bayesian evidence perspective for fitting complex models, including neural networks [MacKay1992]. It does not mean that the largest evidence always identifies the true model: it is conditional on the kernel family being considered and may have several local maxima.

If a hyperparameter is estimated from the data and then treated as exact, its uncertainty is ignored. Fully Bayesian inference would integrate over hyperparameters, which is usually more expensive. Optimizing a single parameter setting is called empirical Bayes or type-II maximum likelihood.

Bayesian quadrature: an integral with uncertainty

An integral is a linear functional of an unknown function. If \(f\) is a GP, then

\[I=\int_a^b f(x)\,dx\]

is normally distributed, provided the required integrals exist. Let \(b_i=\int_a^b k(x,x_i)\,dx\) be the covariance between each observation and the integral, and let \(q=\int_a^b\int_a^b k(x,x')\,dx\,dx'\) be the prior variance of the integral. For a zero-mean prior, GP conditioning gives:

\[I\mid X,y\sim\mathcal N\!\left(b^\mathsf TA^{-1}y,\ q-b^\mathsf TA^{-1}b\right).\]

The mean is a weighted sum of evaluated function values. Unlike an ordinary quadrature rule, the weights are determined by the kernel and sample locations. The posterior variance quantifies remaining uncertainty under the chosen GP model.

How the trapezoidal rule arises from a function prior

A revealing special case is Brownian motion \(B(t)\) with \(B(0)=0\) and kernel \(k(s,t)=\min(s,t)\). Given the function values at endpoints \(a\) and \(b\), the conditional mean between them is the straight line joining those values. The remaining uncertainty has the shape of a Brownian bridge:

\[\mathbb E[B(t)\mid B(a),B(b)]=B(a)+\frac{t-a}{b-a}[B(b)-B(a)],\qquad \operatorname{Var}(B(t)\mid B(a),B(b))=\frac{(t-a)(b-t)}{b-a}.\]

Across adjacent intervals, integrating the conditional mean gives the sum of trapezoid areas. The trapezoidal rule is therefore a posterior mean under this prior, not an assumption-free answer. Even with known endpoints, the integral retains nonzero variance.

Brownian bridge between two known function valuesA linear posterior mean joins two observed endpoints. The uncertainty band is zero at the endpoints and widest in between.abMean: line · uncertainty: Brownian bridge
The conditional mean integrates to the trapezoid area. The band reminds us that the result remains uncertain.

Bayesian quadrature in the browser

This example integrates a synthetic, analytically known function on \([0,1]\). The model uses an SE kernel and numerical Simpson quadrature for the covariance integrals.

The function is known here so the estimate can be checked. In real applications, the exact integral is precisely what we do not know.
Exact demo integral—
GP posterior · mean ± 1.96 SD—
Trapezoidal rule—

The trapezoidal rule can arise as the posterior mean for certain spline priors with noiseless boundary observations. This connects classical numerical methods to probabilistic inference and shows why every numerical answer embeds assumptions about unevaluated locations [OHagan1991].

From regression to probabilistic numerics

For a differential equation, the solution itself is an unknown function. One can place a GP prior on the solution and treat the equation as additional information: candidate trajectories whose derivatives violate the equation receive less posterior support. This makes it possible to propagate uncertainty from a numerical solution into downstream calculations [HennigHauberg2014].

Numerical computation as probabilistic inferenceA numerical problem and finitely many evaluations produce a distribution over solutions, whose uncertainty can be propagated.Equationand boundary dataFinite queriesfunction evaluationsPosteriorsolution + uncertaintyNext steppropagate beliefUncertainty reflects incomplete computation and model assumptions, not physical measurement noise.
Probabilistic numerics tracks uncertainty in a computed quantity so downstream decisions can account for it.

What the band knows—and what it does not

A GP posterior is exact within the chosen model. A narrow uncertainty band does not prove that the model matches reality. An SE kernel cannot represent a discontinuity if it only allows very smooth curves. Missing trends, heteroscedasticity, or a poorly chosen input scale can also distort predictions.

Do not confuse three kinds of uncertaintyEpistemic uncertainty comes from limited data; measurement noise persists even with more data; model error arises when the mean, kernel, or noise model is inappropriate. Under its assumptions, the usual GP band quantifies the first two, but not automatically model error.

The interval \(\mu_*(x)\pm1.96\sigma_*(x)\) is a pointwise 95% interval at a fixed \(x\). It does not mean that an entire random function lies inside the band everywhere with 95% probability. Check coverage on held-out data and compare predictions across plausible kernels and parameter settings.

For \(n\) observations, an exact dense Cholesky factorization costs \(O(n^3)\) time and \(O(n^2)\) memory. Large data sets require approximations such as inducing points, structured covariance matrices, or state-space models. Solve linear systems with Cholesky factorization rather than forming \(A^{-1}\); a small jitter can stabilize rounding errors, but it is not the same as physical measurement noise.

In practice, ask: What quantity is unknown? Which trajectories are plausible? What does the sensor actually measure? How are kernel parameters chosen? Do predicted intervals agree with independent data?

Original sources

The derivations and figures in this chapter are original. These primary papers establish the GP models and probabilistic numerical methods used here.