A handful of measurements can support countless curves. Gaussian process regression turns that ambiguity into a probability distribution over functions, showing both what the data suggest and how uncertain each prediction remains.
This chapter derives the method from Bayesian conditioning and covariance, then builds up to kernels, posterior prediction, and model selection. Interactive labs let you see how assumptions about smoothness and noise change a fit, and how the same framework assigns uncertainty to an integral.
Many curves fit the same observations
Suppose an unknown function has been measured at three locations. A straight line may pass through those points, but so may many other curves. Between the measurements, and beyond their range, infinitely many trajectories remain possible. The data alone do not determine a unique answer.
A conventional regression method usually returns one preferred curve. A GP instead describes a probability distribution over functions. At each input it gives a mean and uncertainty, while also describing how predictions at different inputs relate to one another.
Probability describes uncertainty
A random variable need not represent physical randomness. A fixed but unknown quantity can also be described by a probability distribution. For continuous variables, a density is not a point probability: the probability of an interval is the area under the density.
The mean summarizes a distribution's location; its variance describes its spread. When evidence supports a hypothesis, Bayes' rule updates its distribution:
Here, \(p(f)\) is the prior over functions, \(p(y\mid f)\) the observation model, and \(p(f\mid y)\) the posterior. The denominator normalizes the result. This view keeps the observations separate from the assumptions used to interpret them.
The normal distribution as a building block
A normal distribution is determined by its mean \(\mu\) and standard deviation \(\sigma\). It is useful for GP regression because linear combinations of jointly normal variables remain normal, and their conditional distributions have closed-form expressions.
For a vector of function values, knowing each variance is not enough; we also need the covariance between locations. O'Hagan gave an early Bayesian treatment of curve fitting and prediction using a prior distribution over functions [OHagan1978].
Covariance connects function values
Covariance \(\operatorname{Cov}(A,B)\) measures whether two quantities tend to lie above or below their means together. Positive values indicate aligned deviations; negative values indicate opposing ones. Zero covariance implies independence only when the variables are jointly Gaussian.
The covariances of function values at \(x_1,\ldots,x_n\) form a matrix \(K\), with \(K_{ij}=\operatorname{Cov}(f(x_i),f(x_j))\). A valid covariance matrix is symmetric and positive semidefinite: for every vector \(a\), \(a^\mathsf TKa\ge0\), because this expression is the variance of the linear combination \(a^\mathsf Tf\).
Observing means conditioning
For two standardized, jointly normal variables \(A\) and \(B\) with correlation \(\rho\):
The observation shifts the mean of \(A\) according to \(b\) and \(\rho\), and reduces its variance. If \(\rho=0\), then \(B\) tells us nothing about \(A\). As \(|\rho|\) approaches one, \(A\) becomes nearly determined. For a jointly Gaussian vector split into unobserved values \(u\) and observed values \(v\), the same rule is:
Bayesian linear regression provides the bridge
A line can be written as a weighted sum of features, for example \(f(x)=w_0+w_1x\). Instead of treating its weights as unknown but fixed numbers, place a Gaussian distribution over them. Each possible set of weights then defines a possible line.
After observing data, the weight posterior remains Gaussian. With design matrix \(\Phi\):
A new function value is a linear projection of this Gaussian vector, so it is Gaussian too [WilliamsRasmussen1995]:
Next, let the number of features grow very large, even infinite, without explicitly storing their weights.
The kernel hides the feature space
The covariance between two function values in a linear weight model is the inner product of their features, weighted by the prior covariance of the weights:
If all computations depend only on these inner products, we no longer need to calculate the features individually. Any function \(k(x,x')\) that produces a positive semidefinite covariance matrix for every finite set of inputs can be used directly as a kernel.
The squared-exponential kernel (SE, also called the RBF kernel) in one dimension is
The amplitude \(\sigma_f\) controls typical function magnitude; the length scale \(\ell\) controls how quickly correlation decays with distance. A large \(\ell\) favors slowly varying functions. The SE kernel is extremely smooth, which is a strong modeling assumption.
| Kernel | Covariance at distance r | Typical assumption |
|---|---|---|
| SE / RBF | \(\sigma_f^2e^{-r^2/(2\ell^2)}\) | Very smooth functions |
| Matérn 3/2 | \(\sigma_f^2(1+\sqrt3r/\ell)e^{-\sqrt3r/\ell}\) | Rougher, once-differentiable paths |
| Periodic | \(\sigma_f^2e^{-2\sin^2(\pi r/p)/\ell^2}\) | Repeating patterns with period p |
A Gaussian process is a distribution over functions
A function has infinitely many possible values. A Gaussian process avoids storing an infinite table: for every finite selection of inputs, the corresponding function values are jointly Gaussian, and those distributions must be mutually consistent.
The mean function \(m(x)\) sets the average trajectory; the kernel \(k(x,x')\) sets variances and dependencies. A sampled curve is one realization from the prior, not a “mean plus a little noise.”
Deriving the GP regression formula
We observe \(y_i=f(x_i)+\epsilon_i\), with independent Gaussian noise \(\epsilon_i\sim\mathcal N(0,\sigma_n^2)\). The noise variance is added only to the diagonal of the covariance matrix:
A new latent function value \(f_*=f(x_*)\) is jointly Gaussian with the observations. Applying the conditioning rule gives:
The first term shifts the prediction according to the observation residuals. The second reduces uncertainty to the extent that the new input is correlated with the training points. For a new noisy measurement, add \(\sigma_n^2\) to the latent posterior variance. Uncertainty about \(f(x_*)\) and about a new observation are not the same.
Interactive GP lab
Change the kernel, length scale, signal amplitude, noise level, and number of observations. The synthetic example has a known underlying function; the GP model sees only the selected measurements.
Regression and covariance
Hyperparameters are uncertain too
The kernel contains parameters such as length scale and amplitude. They can be set using domain knowledge, selected by cross-validation, or optimized using the marginal likelihood. With Gaussian noise, its logarithm is:
The first term rewards data fit; the determinant term penalizes, roughly speaking, models that spread their probability mass over too many trajectories. MacKay developed this Bayesian evidence perspective for fitting complex models, including neural networks [MacKay1992]. It does not mean that the largest evidence always identifies the true model: it is conditional on the kernel family being considered and may have several local maxima.
If a hyperparameter is estimated from the data and then treated as exact, its uncertainty is ignored. Fully Bayesian inference would integrate over hyperparameters, which is usually more expensive. Optimizing a single parameter setting is called empirical Bayes or type-II maximum likelihood.
Bayesian quadrature: an integral with uncertainty
An integral is a linear functional of an unknown function. If \(f\) is a GP, then
is normally distributed, provided the required integrals exist. Let \(b_i=\int_a^b k(x,x_i)\,dx\) be the covariance between each observation and the integral, and let \(q=\int_a^b\int_a^b k(x,x')\,dx\,dx'\) be the prior variance of the integral. For a zero-mean prior, GP conditioning gives:
The mean is a weighted sum of evaluated function values. Unlike an ordinary quadrature rule, the weights are determined by the kernel and sample locations. The posterior variance quantifies remaining uncertainty under the chosen GP model.
How the trapezoidal rule arises from a function prior
A revealing special case is Brownian motion \(B(t)\) with \(B(0)=0\) and kernel \(k(s,t)=\min(s,t)\). Given the function values at endpoints \(a\) and \(b\), the conditional mean between them is the straight line joining those values. The remaining uncertainty has the shape of a Brownian bridge:
Across adjacent intervals, integrating the conditional mean gives the sum of trapezoid areas. The trapezoidal rule is therefore a posterior mean under this prior, not an assumption-free answer. Even with known endpoints, the integral retains nonzero variance.
Bayesian quadrature in the browser
This example integrates a synthetic, analytically known function on \([0,1]\). The model uses an SE kernel and numerical Simpson quadrature for the covariance integrals.
The trapezoidal rule can arise as the posterior mean for certain spline priors with noiseless boundary observations. This connects classical numerical methods to probabilistic inference and shows why every numerical answer embeds assumptions about unevaluated locations [OHagan1991].
From regression to probabilistic numerics
For a differential equation, the solution itself is an unknown function. One can place a GP prior on the solution and treat the equation as additional information: candidate trajectories whose derivatives violate the equation receive less posterior support. This makes it possible to propagate uncertainty from a numerical solution into downstream calculations [HennigHauberg2014].
What the band knows—and what it does not
A GP posterior is exact within the chosen model. A narrow uncertainty band does not prove that the model matches reality. An SE kernel cannot represent a discontinuity if it only allows very smooth curves. Missing trends, heteroscedasticity, or a poorly chosen input scale can also distort predictions.
The interval \(\mu_*(x)\pm1.96\sigma_*(x)\) is a pointwise 95% interval at a fixed \(x\). It does not mean that an entire random function lies inside the band everywhere with 95% probability. Check coverage on held-out data and compare predictions across plausible kernels and parameter settings.
For \(n\) observations, an exact dense Cholesky factorization costs \(O(n^3)\) time and \(O(n^2)\) memory. Large data sets require approximations such as inducing points, structured covariance matrices, or state-space models. Solve linear systems with Cholesky factorization rather than forming \(A^{-1}\); a small jitter can stabilize rounding errors, but it is not the same as physical measurement noise.
In practice, ask: What quantity is unknown? Which trajectories are plausible? What does the sensor actually measure? How are kernel parameters chosen? Do predicted intervals agree with independent data?
Original sources
The derivations and figures in this chapter are original. These primary papers establish the GP models and probabilistic numerical methods used here.
- [WilliamsRasmussen1995]Williams, C. K. I., & Rasmussen, C. E. (1995). Gaussian Processes for Regression. Advances in Neural Information Processing Systems 8, 514–520.
- [OHagan1978]O'Hagan, A. (1978). Curve Fitting and Optimal Design for Prediction (with Discussion). Journal of the Royal Statistical Society: Series B, 40(1), 1–42.
- [MacKay1992]MacKay, D. J. C. (1992). A Practical Bayesian Framework for Backpropagation Networks. Neural Computation, 4(3), 448–472.
- [OHagan1991]O'Hagan, A. (1991). Bayes–Hermite Quadrature. Journal of Statistical Planning and Inference, 29(3), 245–260.
- [HennigHauberg2014]Hennig, P., & Hauberg, S. (2014). Probabilistic Solutions to Differential Equations and their Application to Riemannian Statistics. Proceedings of AISTATS, PMLR 33, 347–355.