Take a set of points scattered over a plane and a horizontal line, connected to every point by a vertical segment. Move your mouse up and down over the diagram below to slide the line and watch the accumulated distance change; click anywhere on it to jump back to the optimum:
A natural first instinct is to minimize the sum of the squared distances, since that leads to a clean derivative. But squared error has a well-known weakness: two points sitting close together and one point far away will pull the optimum toward the distant outlier, because a large error is penalized quadratically. The sum of squared distances is minimized by the line's value being the mean, and the mean is exactly the statistic that a single outlier can drag arbitrarily far. So the squared distance is the wrong objective here — the goal is to minimize the sum of the plain, unsquared distances.
Solving the Optimization Problem
Only the y-coordinates of the points matter for this problem, so let
\[Y=\{y_1,y_2,\dots,y_n\}\subset\mathbb{R}.\]
The goal is to find a value \(\hat{y}\) that minimizes the sum of the absolute distances to every point in \(Y\):
\[\underset{\hat{y}}{\operatorname{argmin}}\sum\limits_{y\in Y}|y-\hat{y}|.\]
Setting the derivative of the objective to zero would require solving
\[\sum\limits_{y\in Y}\operatorname{sgn}(y-\hat{y})=0,\]
which is not directly usable, since \(\operatorname{sgn}\) is neither differentiable at \(0\) nor invertible in closed form. Trying every candidate \(\hat{y}\) by brute force would work, but there is a much cheaper route: split \(Y\) into the points at or below \(\hat{y}\) and the points strictly above it,
\[P=\{\,p\in Y:p\le\hat{y}\,\},\qquad Q=\{\,q\in Y:q>\hat{y}\,\}.\]
With this split, the objective function \(f(\hat{y})\) can be rewritten without any absolute values at all:
\[ \begin{array}{rl} f(\hat{y}) &= \displaystyle\sum\limits_{y\in Y}|y-\hat{y}| \\[6pt] &= \displaystyle\sum\limits_{p\in P}|p-\hat{y}| + \sum\limits_{q\in Q}|q-\hat{y}| \\[6pt] &= \displaystyle-\sum\limits_{p\in P}(p-\hat{y}) + \sum\limits_{q\in Q}(q-\hat{y}) \\[6pt] &= \displaystyle-\Bigl(\sum\limits_{p\in P}p-\sum\limits_{p\in P}\hat{y}\Bigr) + \Bigl(\sum\limits_{q\in Q}q-\sum\limits_{q\in Q}\hat{y}\Bigr) \\[6pt] &= \displaystyle-\sum\limits_{p\in P}p + |P|\hat{y} + \sum\limits_{q\in Q}q - |Q|\hat{y}. \end{array} \]
Every \(p\in P\) satisfies \(p\le\hat{y}\), so \(|p-\hat{y}|=\hat{y}-p=-(p-\hat{y})\), which is where the sign flip in the third line comes from; every \(q\in Q\) satisfies \(q>\hat{y}\), so \(|q-\hat{y}|=q-\hat{y}\) needs no sign flip. With \(f(\hat{y})\) now a plain linear function of \(\hat{y}\), the derivative is trivial:
\[ \begin{array}{rl} \displaystyle\frac{\partial}{\partial\hat{y}}f(\hat{y}) &= \displaystyle\frac{\partial}{\partial\hat{y}}\Bigl(-\sum\limits_{p\in P}p + |P|\hat{y} + \sum\limits_{q\in Q}q - |Q|\hat{y}\Bigr) \\[6pt] &= |P|-|Q|. \end{array} \]
Setting this to zero gives the optimality condition:
\[0=|P|-|Q|\quad\Longleftrightarrow\quad|P|=|Q|.\]
In other words, the sum of absolute distances is minimized exactly when \(\hat{y}\) splits \(Y\) into two halves of equal size — the same number of points at or below \(\hat{y}\) as strictly above it. If you remember your stats class, that condition has a familiar name:
\[\hat{y}=\widetilde{y},\]
the median. Spelled out: the value that minimizes the sum of absolute distances to a set of points is their median, not their mean. For an odd number of points, \(|P|=|Q|\) pins down a single value, the middle order statistic. For an even number of points, any value between the two middle order statistics keeps \(|P|=|Q|\) and therefore achieves the same minimal sum — the objective is flat over that whole interval, which is exactly why the median of an even-sized sample is conventionally defined as the average of those two middle values, one particular point in an entire interval of equally optimal solutions.
Go back to the diagram and click anywhere on it: the line jumps straight to the median and the accumulated distance drops to its minimum.