Strip every application in this book to its mathematical bones and a great many of them reduce to the same question: given a vector of measurements x drawn from one of two populations, decide which population it came from. Call the populations classes and label them y = +1 and y = -1. The simplest possible answer to the discrimination problem is also, historically, the first one anybody worked out in full generality: compute a single number from x by a weighted sum of its coordinates, compare that number to a threshold, and let the sign of the comparison be the decision. Formally, a linear discriminant function is a map g(x) = w·x + b, where w is a vector of coefficients ("weights") and b a scalar ("bias" or "threshold"), and the decision rule is ŷ = sign(g(x)). The set of points where g(x) = 0 is a hyperplane — a line in two dimensions, a plane in three, an affine subspace of one dimension less than the input in general — and it is this hyperplane, not the vector w or the scalar b individually, that does the classifying: everything on one side is called positive, everything on the other negative. Two classes of vectors are called linearly separable if some choice of w and b puts every member of one class on the positive side and every member of the other on the negative side with no exceptions. Nothing so far has said how w and b are to be found. That question split, almost immediately, into two research programs that did not speak to each other for decades, and both of them are this chapter's subject.
The first program treated the coefficients as parameters of a statistical model to be estimated, not as anything to be searched for by trial and error. If the two classes are generated by known probability densities f and g, the optimal decision — the one minimizing the probability of error — is a likelihood ratio test on f(x)/g(x); this had been settled by the Neyman-Pearson lemma years before anyone applied it to classification. Evelyn Fix and J. L. Hodges, writing in 1951 on the harder problem of discrimination when the densities are not known at all, paused to name the intermediate case, in which the densities are assumed to belong to a known family with unknown parameters, and identified its canonical instance: "The most familiar example of this process is the linear discriminant function (Fisher, 1936)." In Fisher's construction, as Fix and Hodges describe it, "it is (tacitly) assumed that F and G are p-variate normal distributions having the same (unknown) covariance matrix, and unknown expectation vectors. The two expectation vectors and the covariance matrix are estimated from the samples, and the likelihood ratio procedure is then employed, using the estimated values as though they were known to be correct." Work through the algebra of that likelihood ratio under a shared covariance matrix and the quadratic terms cancel; what remains is exactly a linear function of x. Fisher's linear discriminant, arrived at from an assumption about how the data were generated, and Rosenblatt's perceptron, arrived at twenty-two years later from no distributional assumption at all, were computing the same kind of object — a weight vector and a threshold — by irreconcilably different routes.
The second program asked no question about how the data were generated and made no reference to a likelihood at all. Frank Rosenblatt, describing the perceptron in 1958, was explicit that its learning procedure was meant to model trial-and-error learning rather than estimation: in the general case the experimenter "does not force the system" to produce the desired output, but merely applies positive reinforcement when the response is correct and negative reinforcement when it is wrong. Translated into the notation already on the table, the rule is an update on mistakes alone: present an example x with true label y; if sign(w·x + b) already agrees with y, change nothing; if it disagrees, nudge the weight vector toward correctness, w ← w + yx (and b similarly), and move to the next example. Nothing here refers to a covariance matrix, a class-conditional density, or a sample mean — the procedure only ever looks at whether the current hyperplane is presently wrong about the point in front of it, and corrects in that point's favor. Whether such a myopic, one-example-at-a-time rule converges to a hyperplane that is right about every point at once, and how many corrections that can possibly take, was not settled by Rosenblatt's reinforcement account of the brain; it took a separate line of argument, due to Block and to Novikoff in the early 1960s, to show that it does, and to bound the number of mistakes by a single quantity — the size of the margin by which the classes are separated, relative to the length of the input vectors. That argument, not the biological framing that motivated it, is what turned the perceptron into a machine-learning algorithm rather than a psychological model, and it is the subject of the next section.
The question s1 left open — whether a rule that corrects only the mistake in front of it can ever be trusted to find a hyperplane that is right about every point at once — was answered independently by H. D. Block and A. B. J. Novikoff in 1962, and the answer has survived essentially unchanged into every later restatement of it, including Freund and Schapire's 1999 re-derivation for the voted-perceptron algorithm. It is convenient to absorb the bias into the weight vector before stating it: append a constant coordinate to every input x, so that the threshold b becomes just another entry of an enlarged w, and write the update rule in the homogeneous form used since Rosenblatt — start from v = 0, predict ŷ = sign(v·x), and on a mistake set v ← v + yx. Suppose the training sequence is in fact linearly separable, and separated with room to spare: there exists a unit vector u and a margin γ > 0 such that y(u·x) ≥ γ for every example, and every instance satisfies ‖x‖ ≤ R. The claim, due to Block and Novikoff and reproved by Minsky and Papert in 1969, is that no matter how the examples are ordered, and no matter how many times the sequence is repeated, the number of mistakes the online perceptron can make is bounded — by a quantity that depends only on the margin and the data's scale, not on the dimension of x at all.
Freund and Schapire state the result in exactly this form, crediting it forward through three decades of restatement: "Block (1962), Novikoff (1962) and Minsky and Papert (1969) have shown that if the data are linearly separable, then the perceptron algorithm will make a finite number of mistakes, and therefore, if repeatedly cycled through the training set, will converge to a vector which correctly classifies all of the examples." The precise statement they then reprove reads: "Let ⟨(x1, y1), . . . , (xm, ym)⟩be a sequence of labeled examples with ∥xi∥≤R. Suppose that there exists a vector u such that ∥u∥= 1 and yi(u · xi) ≥γ for all examples in the sequence. Then the number of mistakes made by the online perceptron algorithm on this sequence is at most (R/γ )2." The bound (R/γ)² is the whole content of the theorem: it says that a wide margin relative to the data's radius buys a short learning curve, that a thin margin buys a long one, and that the number of input features plays no part in the story at all — a two-dimensional problem and a ten-thousand-dimensional one obey the identical bound provided R and γ are the same.
The proof tracks two quantities through the sequence of mistakes and squeezes them against each other. Let v_k be the weight vector in force just before the k-th mistake, so v_1 = 0. Freund and Schapire set it up exactly this way: "Let vk denote the prediction vector used prior to the kth mistake. Thus, v1 = 0 and, if the kth mistake occurs on (xi, yi) then yi(vk · xi) ≤0 and vk+1 = vk + yixi." The first quantity to track is how fast v_k's overlap with the true direction u grows. Because the update always moves in the direction of a correctly-labeled point, each mistake adds at least γ to that overlap: "vk+1 · u = vk · u + yi(u · xi) ≥vk · u + γ. Therefore, vk+1 · u ≥kγ ." So after k mistakes, v_k has grown to overlap u by at least kγ — a quantity that grows linearly in the number of mistakes. The second quantity is how fast v_k's own length can grow, and here the mistake condition — that the point was on the wrong side, yi(vk · xi) ≤ 0 — does the opposite work, capping the growth: "∥vk+1∥2 = ∥vk∥2 + 2yi(vk · xi) + ∥xi∥2 ≤∥vk∥2 + R2. Therefore, ∥vk+1∥2 ≤kR2." The vector's squared length can grow by at most R² per mistake, so after k mistakes ‖v_k‖ is at most √k·R — growing only as the square root. A vector whose overlap with a fixed unit direction grows linearly cannot itself have a length growing only as a square root forever, since overlap can never exceed length: "Combining, gives √ kR ≥∥vk+1∥≥vk+1 · u ≥kγ which implies k ≤(R/γ )2 proving the theorem." The two growth rates — linear against root-k — can only coexist for finitely many steps, and solving √k·R ≥ kγ for k hands back the bound directly. Nothing in the six lines refers to the dimension of x, the order in which examples arrive, or how many times the same example is repeated; the only quantities that enter are the margin and the radius.
Two things are worth separating out, because the same two names attach to both and it is easy to run them together. Minsky and Papert's 1969 book, on the strength of its later reputation rather than anything in the pages read for this book — Minsky and Papert's own text is not among the sources examined here, and what follows is received historical knowledge rather than a claim checked against their primary text — is remembered chiefly for the opposite kind of result: an argument that a single-layer perceptron cannot represent a function like exclusive-or, because no hyperplane separates that function's positive and negative cases at all. That is a statement, if it is right, about which functions are reachable in principle; the convergence bound reproved above, which Freund and Schapire do credit to the same 1969 book and which this book's sources do support directly, is a statement about how fast the algorithm finds a hyperplane given that one exists. The two are routinely bundled together under Minsky and Papert's name, but it is only the convergence half — attested here in Freund and Schapire's own restatement — that this section has actually been about. Distinguishing them also settles, cleanly, whether the bound belongs to the book's recurring question about hardware ceilings versus mathematical ones. (R/γ)² is not a number that improves with a faster machine — it does not care whether the dot product at each mistake is computed by hand, by a 1958 vacuum-tube machine, or by a bank of 2026 GPUs; it is a statement about how many mistakes any implementation of this update rule must make before it stops erring, full stop. What a faster machine buys is only the ability to pay for those mistakes faster — which is exactly why the bound, not the hardware, is the right unit of history here, and why the next section can run Rosenblatt's own experiment and ask how the bound and the era's actual machine time traded against each other.
Rosenblatt never left behind the numbers this book most wants: no surviving account states how many hours the Mark I Perceptron's motor-driven potentiometers spent grinding through a training set, or how that time scaled with the number of mistakes the bound of the previous section predicts. What survives instead is the next best thing — a deliberate, decades-later rerun of an experiment in the same spirit, on a scale Rosenblatt could not have had. In 1999, Yoav Freund and Robert Schapire set out explicitly to repeat a benchmark from four years earlier, testing among their conditions the perceptron algorithm in its original, undressed form: no kernel, no lifted dimension, just Rosenblatt's 1958 update rule applied directly to raw pixels. Their own framing makes the lineage explicit.
Freund and Schapire describe the setup precisely: "We repeated some of the experiments performed by Cortes and Vapnik (1995) on the use of SVM on the problem of classifying handwritten digits." The dataset was NIST OCR, and it dwarfed anything Rosenblatt trained on: "The dataset consists of 60,000 training examples and 10,000 test examples. We treat each image as a vector in R784" — each digit a 28×28 grid of grey values, flattened to a point in a 784-dimensional space. Onto this they applied a polynomial kernel of degree d, which maps each point into a much higher-dimensional space before the perceptron's ordinary dot-product update runs inside it. Degree d = 1 does nothing — it leaves the raw pixel vector alone. Rosenblatt's original algorithm was therefore not a special case to be approximated; it was one of the six conditions actually run: "We performed experiments with polynomial kernels for dimensions d = 1 (which corresponds to no expansion) up to d = 6."
The result at d = 1 is the clearest illustration in this book's whole run of what the (R/γ)² bound actually costs when γ is small. Raw handwritten pixels are not a well-separated problem: "the data are not linearly separable which means that the perceptron algorithm tends to make many mistakes." Every one of those mistakes is a pass through 784-dimensional data and a weight update, on real 1998 hardware, and the mistakes did not stay a manageable number. By their own tally of mistakes made across the ten one-against-all digit classifiers, training at d = 1 had accumulated 223,657 mistakes by the tenth epoch — against 32,451 at d = 2 over the same ten epochs, a sevenfold difference produced by nothing except the shape of the decision boundary. The wall-clock consequence was blunt: "This is why, for d = 1, we could not even complete a run out to 30 epochs but had to stop at T = 10 (after about six days of computation). In comparison, for d = 2, we can run 30 epochs in about 25 hours, and for d = 5 or 6, a complete run takes about 8 hours. (All running times are on a single SGI MIPS R10000 processor running at 194 MHZ.)" Lifting the data into a higher-dimensional kernel space did not slow the machine down, as raising the dimension would naively suggest — it sped it up, because it bought separability, and separability is what the mistake bound is actually priced in. Freund and Schapire's own reading of the gap between the two runs makes the point directly: "The most significant improvement in performance is clearly between d = 1 and d = 2."
Put the two halves of this experiment through the question the last section left open. The six days is pure hardware, and nothing about it is durable: a single SGI workstation clocked at 194 MHz is not a fact about learning, it is a fact about 1998, and a 2026 machine erases the difference between six days and eight hours without anyone noticing there was ever a gap. But the ratio behind it — 223,657 mistakes at d = 1 against 32,451 at d = 2, on the identical data, the identical labels, the identical stopping point — is not a hardware fact at all. It is the (R/γ)² bound made visible: a small or vanishing margin on raw, unlifted pixels forces a long run of corrections, and no datacenter changes how many mistakes a fixed margin permits before the theorem stops binding. What the extra dimensions bought was not speed directly; it was margin, and margin is what the bound actually taxes. The honest caveat belongs here too. Even at its best, the perceptron did not catch the method it was rerunning against: Cortes and Vapnik's support vector classifier, on the same kernel, was "achieving a test error rate of 1.1% for d = 4," while the voted perceptron's best results hovered further above it. Freund and Schapire's own summary does not pretend otherwise — the perceptron's numbers were "comparable, though still inferior, to those obtainable with support-vector machines" — and the chapter's arc closes on exactly that asymmetry. Rosenblatt's algorithm was never the most accurate way to find a separating hyperplane; the mistake-bound never promised accuracy, only a ceiling on how much correction the machine would be made to pay for. What the rerun of the original experiment shows, forty years on and on data two orders of magnitude larger than Rosenblatt ever saw, is that the ceiling was still there, still doing exactly what Block and Novikoff said it would — and still, as it always was, indifferent to which machine was made to pay it.
Freund and Schapire's rerun settles the qualitative claim but leaves the (R/γ)² bound itself unmeasured — they report mistake counts and running times, not the margins that are supposed to be driving them. That gap can be closed directly, on hardware neither Rosenblatt nor Freund and Schapire had reason to expect: a 2020s laptop, sklearn, and a dataset in the same family as theirs but not the same one. In place of NIST OCR's 60,000 28×28 training images, the rerun below uses scikit-learn's bundled digits set — 1,797 handwritten digits, each an 8×8 grid of grey levels (a coarser scan than NIST's, chosen because it ships with the library and needs no separate download), split into all forty-five one-against-one pairs. This substitution should be stated plainly: it is modern, low-resolution stimuli standing in for Rosenblatt's photocells and for Freund and Schapire's NIST scans, not a replication of either dataset. What it preserves is the thing the bound is actually about — a perceptron run to convergence on real images, with a margin that can be measured rather than assumed.
The protocol: for each of the 45 digit pairs, run Rosenblatt's update rule exactly as Novikoff analyzed it — augment each image vector with a constant bias coordinate, initialize the weight vector at zero, and on each mistake add the misclassified example's (signed) vector to the weights — sweeping the training examples in epochs until a full epoch produces zero corrections. Separately, fit a hard-margin linear support-vector classifier (scikit-learn's SVC, kernel='linear', C=10⁴) to the same pair, and from its weight vector compute the achieved margin γ = min_i y_i(w·x_i + b)/‖w‖ and the radius R = max_i‖x_i‖, the two quantities Novikoff's theorem combines into a bound of (R/γ)² mistakes. Executed exactly this way, the perceptron converged on every one of the 45 pairs — digits at 8×8 resolution turn out to be linearly separable pair by pair, so the run measures the bound's tightness rather than testing whether it fails to converge. The two extremes make the shape of the result vivid: the pair 6-versus-7, the easiest, converged in two epochs (four mistakes in the first pass, none in the second), with a measured margin γ = 11.78 and a bound of (R/γ)² ≈ 37 — the theorem gave itself roughly nine times the mistakes actually needed. The pair 1-versus-8, the hardest of the 45, took 25 epochs and 262 mistakes to converge, with a measured margin of only γ = 1.80 against the same order of R, driving the bound up to (R/γ)² ≈ 1,824. Across all 45 pairs the bound never once failed — every measured mistake count sat below its own (R/γ)² — and the correlation between the bound and the actual mistakes made, measured on this run, was a Pearson r = 0.973 (r = 0.975 between 1/γ² alone and mistakes, since R varies only mildly across pairs relative to γ, so the two correlations are close but not identical). Small margin predicted a hard training run, and small margin is what produced one.
The same run also reproduces, in miniature, the exact effect Freund and Schapire reported for polynomial degree: lifting the hardest pair into a higher-dimensional feature space shrinks the mistake count by buying margin, not by doing anything to the hardware. Taking the single worst pair from the sweep, 1-versus-8, and running a kernel perceptron on it — the dual form of the same update rule, with a polynomial kernel K(x,z) = (x·z + 1)ᵈ standing in for the raw dot product — at degree d = 1 (no expansion, identical to the plain perceptron above) the run needed 25 epochs and 262 mistakes to converge, with a kernel-space margin of 1.80. At d = 2 it converged in 13 epochs on 117 mistakes, with a kernel-margin of 237.9. At d = 3 it converged in 6 epochs on 65 mistakes, margin 18,547.2. Fewer mistakes, fewer epochs, larger measured margin, moving in lockstep exactly as the bound says they must — on a pair of digits and a laptop, not NIST OCR and an SGI R10000, which is the point: the qualitative finding Freund and Schapire published in 1999 is not an artifact of their particular machine or their particular scan of handwriting, since it reappears when both are replaced.
Put next to Freund and Schapire's own numbers, this measured rerun closes the gap their 1999 paper left open. They showed that mistakes ballooned when margin was small and running time ballooned with it — 223,657 mistakes and six days at d = 1 against 32,451 mistakes and twenty-five hours at d = 2 — but margin itself was never in their tables, only inferred from the shape of the result. Measured directly, on a different dataset and a different machine, the same inference holds exactly: γ = 1.80 against γ = 237.9 against γ = 18,547.2, tracking 262, 117, and 65 mistakes in the same order, and across the full sweep of 45 pairs the bound (R/γ)² sat above the true mistake count every single time, correlating with it at r = 0.973 (against r = 0.975 for 1/γ² alone). None of that is a hardware fact. A 2026 datacenter runs all 45 pairs and all three kernel degrees in well under a second — the six days and the twenty-five hours have already evaporated once, from 1998 to 1999-era SGI hardware to whatever ran this book's own rerun, and they will evaporate again. What does not evaporate is the shape of the relationship: γ small still means many mistakes, γ large still means few, and the constant of proportionality is (R/γ)², not FLOPS. That is the finding this chapter set out to isolate, and it is now a measured one rather than an inferred one.
Volume II, Chapter 1 ended with a bound — (R/γ)² — that has nothing to do with hardware and everything to do with geometry: it counts the mistakes an online perceptron must make before a linearly separable problem is solved, and it holds regardless of what computes the updates. But the bound presupposes separability. Rosenblatt's threshold unit computes ŷ = sign(w·x − θ) for a single weight vector w and threshold θ, and the set of inputs it can classify correctly is exactly a halfspace of the input space — one side of a hyperplane. Not every classification problem is a halfspace problem. The textbook case, small enough to check by hand, is exclusive-or on two binary inputs: the positive cases (0,1) and (1,0) and the negative cases (0,0) and (1,1) cannot be separated by any single straight line in the plane, because the positive pair and the negative pair each straddle the other. No choice of w and θ fixes this; the function is simply not in the class of functions a single threshold unit can express. The fix is architectural rather than algorithmic: interpose a layer of threshold units between the input and the output, let each hidden unit compute its own halfspace, and let the output unit combine those halfspaces. Two hidden units suffice for XOR — one can implement OR, the other AND-NOT, and the output unit ANDs their outputs — and nothing in principle stops a hidden layer from bending an arbitrarily complicated decision surface out of enough such pieces. What stood between this observation and a working multilayer classifier for a further decade and a half was not a representational limit but a procedural one, and stating it precisely is the business of this section.
The primary source used here for the architecture is Robert Hecht-Nielsen's 1989 formalization, written three years after Rumelhart, Hinton, and Williams's own paper and offered as a rigorous restatement of it; the original PDP-group chapter is not among the texts examined for this book, so Hecht-Nielsen's notation stands in for it, and any place this section states an equation rather than quoting one is a modernization of his (partly OCR-damaged) notation, flagged as such. The architecture is "a hierarchical design consisting of fully interconnected layers or rows of processing units," numbered from the input up; the bottom row simply distributes the input vector — "the first layer consists of n fanout processing elements that simply accept the individual components ... of the input vector x and distribute them, without modification, to all of the units of the second row" — and every later row computes from the row beneath it. All the rows strictly between the input and the final, output row are what will matter most for what follows: the network is built to approximate, from a training sample of input–output pairs, "a bounded mapping or function," and each interior unit's output is a smooth, saturating function — a sigmoid — of a weighted sum of the row below it, exactly the logistic squashing function already met in Volume I as the ingredient that makes the whole map differentiable. Computing an input all the way to an output — pushing activity up through the rows in order — is what Hecht-Nielsen calls the forward pass, and on its own it is nothing new: it is the ordinary evaluation of a composition of affine maps and sigmoids, layer after layer, and every one of the historical machines discussed in this chapter could evaluate it in a fraction of a second. The problem this chapter is about begins only once that forward computation is compared against a target and the mismatch has to be turned into a correction — because most of the units that did the computing sit nowhere near the comparison.
Name the difficulty precisely: on the forward pass every hidden unit's output is computable, and the mismatch between the output row and the target vector is computable, but the training signal — the correct output — is only ever supplied to the top row. A hidden unit has no target of its own to be compared against; it has only its contribution, filtered through every unit above it, to an error that is reported three or more layers away. Deciding how much of the blame for a wrong answer belongs to a given hidden unit's weights, when that unit is connected to no target at all, is what the connectionist literature came to call the credit assignment problem, and it is a different kind of difficulty from anything Volume II's first chapter required: the perceptron's error signal (t − y) was already sitting at the unit whose weights needed changing. Multiply the layers and the signal has nowhere obvious to go. The history of solving it is, appropriately, a history of the same idea being lost and refound. Hecht-Nielsen's own account, compiled three years after the fact, lists the rediscoveries: the method "was originally introduced by Bryson and [Ho] in 1969 and independently rediscovered by Werbos in 1974, by Parker in the mid 1980's and by Rumelhart, Williams and other members of the PDP group in 1985" — and Werbos's version sat uncited and unappreciated for over a decade, not because the mathematics in it was wrong but because none of the fields that could have used it were reading the others' journals. What changed in 1985 was not the derivative but the audience: "credit for developing backpropagation into a usable technique, as well as promulgation of the architecture to a large audience, rests entirely with Rumelhart and the other members of the PDP group." The chain rule that solves the credit assignment problem is, as the next section will derive from Hecht-Nielsen's restatement, nothing more exotic than repeated application of calculus a hidden unit at a time — but recognizing that repeated differentiation was the answer, and that it could be organized into an algorithm cheap enough to run on the machines then available, took sixteen years and four independent discoveries.
There is a second reason the credit assignment problem deserves a section of its own rather than a paragraph inside the backpropagation derivation, and it is the reason this book cares about it at all: some of the difficulty is not procedural but mathematical, and would survive any amount of 2026 hardware. Blum and Rivest, writing after backpropagation was already in wide use, asked the question directly — noting that "in practice ... the back propagation algorithm often runs very slowly," and asking "whether there are necessarily intrinsic computational difficulties associated with training neural networks, or whether better training algorithms might exist." Their answer sharpens the contrast with Volume II's first chapter almost exactly. For the single-layer perceptron, "if there exist edge weights so that the network correctly classifies a given training set, then such weights can be found in time guaranteed to be polynomial in n, using linear programming" — the perceptron convergence proof of Chapter 1, restated as a complexity guarantee. For a network with even one hidden layer, that guarantee is gone: Blum and Rivest prove "that no such training algorithm exists for a very simple 2-layer network with only two hidden nodes and a single output node, unless a widely believed complexity-theoretic assumption proves false," namely P = NP. Finding weights that fit the training set — not approximating them, not finding the best ones, just deciding whether a fit exists at all — is NP-complete for a network with a single hidden layer of two units. This is the line the book keeps asking every claimed limitation to cross, and here the claim crosses it. A slow run on a 1988 workstation is a hardware fact, erased by a 2026 datacenter; an NP-completeness proof is not erased by better silicon, because it says the difficulty grows faster than any fixed polynomial in the problem size, on every machine there will ever be. Backpropagation, whose derivation occupies the rest of this chapter, does not contradict Blum and Rivest's result — it sidesteps the question they answered. It does not decide whether a perfect fit exists; it follows a gradient downhill and accepts whatever local optimum it finds, cheaply, one hidden unit's worth of chain rule at a time. That two answers to two different questions — one intractable in the worst case, one merely heuristic — arrived attached to the same architecture is the setup this chapter has to keep straight before it writes down a single derivative.
Put the architecture of the previous section into symbols. For a network with one hidden layer, write the input pattern as x, the hidden unit j's net input as net_j = Σ_i w_ji x_i + θ_j and its output as o_j = f(net_j); the output unit k's net input as net_k = Σ_j w_kj o_j + θ_k and its output as o_k = f(net_k); and let f be the logistic sigmoid, f(a) = 1/(1+e^{-a}), whose derivative is the convenient closed form f'(a) = f(a)(1 - f(a)). (This is a modernization of Hecht-Nielsen's own notation, which numbers units by row and calls the weight from unit j to unit i "w_lij"; his printed formulas survive OCR extraction only patchily, so the equations here are restated rather than quoted, and every place this section states rather than quotes an equation that convention holds.) For a single training pattern p with target vector t, define the error as the summed squared mismatch at the output row, E_p = (1/2)Σ_k (t_k − o_k)^2, and the total error E as the sum of E_p over the training sample — exactly the "mean squared error function" whose differentiability Hecht-Nielsen spends several pages of the same paper proving exists at all before he lets himself compute a derivative of it. Nothing so far is new: it is the forward pass, unit by unit, row by row. What turns this forward computation into a training algorithm is the direction the correction has to run, and Hecht-Nielsen states the shape of it plainly even where his equations do not survive intact: each training cycle "consists of the inputs to the network "bubbling up" from the bottom to the top and then the errors "percolating down" from the top to the bottom." The rest of this section is the arithmetic of that second, downward sweep — the chain rule, applied one row at a time, that the credit-assignment problem of the previous section demanded and that Bryson and Ho, Werbos, Parker, and the PDP group each independently supplied.
Differentiate E_p with respect to a weight feeding the output row first, since there the target is available directly. By the chain rule, ∂E_p/∂w_kj = (∂E_p/∂net_k)(∂net_k/∂w_kj). The second factor is just o_j, from the definition of net_k. Call the first factor δ_k ≡ ∂E_p/∂net_k; for the sum-of-squares error and a sigmoid output unit, δ_k = −(t_k − o_k) f'(net_k), and so ∂E_p/∂w_kj = −(t_k − o_k) f'(net_k) o_j — an error term local to unit k, multiplied by the activation flowing into that weight. This is the perceptron's (t − y) term from Volume II's first chapter, reappearing with a sigmoid-derivative factor attached, and it is computable the moment the output row sees the target: no credit-assignment problem arises here because the "credit" is the discrepancy itself. Now push the same differentiation one row down, to a weight w_ji feeding a hidden unit j, which has no target of its own. The chain rule still applies, but net_j now influences E_p only indirectly, through every net_k that j feeds: ∂E_p/∂net_j = Σ_k (∂E_p/∂net_k)(∂net_k/∂net_j) = Σ_k δ_k w_kj · f'(net_j). Define δ_j ≡ ∂E_p/∂net_j = f'(net_j) Σ_k w_kj δ_k, and the gradient with respect to the hidden weight follows exactly as before: ∂E_p/∂w_ji = δ_j x_i. The hidden unit's δ_j is nothing but a weighted sum of the output row's δ_k's, weighted by the very connections that carried the forward signal up — the error is literally sent back down the same wires the activation went up, scaled by how much each wire mattered and by how steep the hidden unit's own sigmoid was at the operating point. That recursion is the whole of the credit-assignment solution the setup section left open: a hidden unit is not compared to a target, it is compared to the weighted blame handed down from the units it feeds, one row at a time, however many rows there are. Hecht-Nielsen states the resulting guarantee about where this leaves the weights without deriving it in survivable notation: the "generalized delta rule learning law used with the backpropagation neural network has the property that, given any starting point w₀ on the error surface that is not a minimum, the learning law will modify the weight vector w so that F(w) will decrease" — descent, not solution; the gradient tells the network which way is downhill and nothing about whether downhill ends anywhere good.
Chris Bishop's 1992 note restates exactly this recursion in cleaner notation before extending it, and it is worth pausing on his restatement because it fixes the symbols the rest of this section needs. Bishop writes the activation of the ith unit as a nonlinear function of its net input, "the activation zi of the ith unit is a nonlinear function of the input to the unit: zi = f (ai) (2.1) in which the input ai is given by a weighted linear sum of the outputs of other units where wi, is the synaptic weight from unit j to unit i, and Bi is a bias associated with unit i" — z_i and a_i standing in for this section's o and net, w_ij for this section's w_ji, with biases folded into the weight matrix as connections from a unit whose output is fixed at 1. Working through the same chain rule as above, he arrives at the identical hidden-unit recursion and calls it what it is: "This is just the familiar backpropagation equation." His restatement of the output-layer gradient is equally direct — "Note that the first derivatives of the error function are given by the standard expression" δ_j z_i, his δ_j playing the role of this section's δ_k or δ_j depending on the row. Two independent derivations, three years apart and citing different lineages, land on the same arithmetic; that convergence is itself evidence that the chain-rule solution to credit assignment is not a modeling choice but the unique answer once the architecture and the error measure are fixed. Bishop's paper is not really about the first derivative, though. It exists to go one step further, to the second derivatives — the Hessian matrix of E with respect to every pair of weights — and to show they can be computed exactly rather than approximated. The motivation is practical and specific to what a second derivative buys that a first derivative cannot: "Second derivative information has been used to provide a fast" procedure for retraining a network after its training data changes slightly; "Approximations to the Hessian have been used to identify the least significant weights" as the basis of network-pruning methods such as Optimal Brain Damage; and the Hessian has been "used by MacKay (1991) for Bayesian estimation of regularization parameters" and for assigning error bars to a trained network's outputs. Each of these uses needs the curvature of the error surface, not just its slope, and each had previously made do with an approximation — until, as Bishop reports of MacKay's own finding, "MacKay found that the approximation scheme of Le Cun et al. (1990) was not sufficiently accurate and therefore included off-diagonal terms in the approximation" — the diagonal-only shortcut used for pruning was not good enough for Bayesian use, and a method for the exact matrix was worth having.
The mechanics of Bishop's exact algorithm follow the same two-sweep shape as ordinary backpropagation, run twice over. A first forward pass computes every unit's activation z_i as usual; a second forward pass propagates a set of sensitivity terms g_li — how the net input of a later unit l responds to a change in the net input of an earlier unit i — using a recursion built from the same weights and the same f'(a) that the ordinary forward pass already touched. Then a first backward pass recovers the familiar first-derivative δ_n at every unit, exactly as in the previous derivation; a second backward pass propagates a further set of terms b_ni, combining the second derivative of the activation function f''(a) with the g_li already computed and the δ's already backpropagated. The Hessian entry for any pair of weights is then assembled, pattern by pattern, from these forward-propagated and backward-propagated quantities and summed over the training sample — an algorithm Bishop states plainly is exact, not approximate: extending it costs, in his own accounting, at most twice as many forward-and-backward propagations per pattern as there are hidden-or-output units in the network — but each of those sweeps involves work that scales with the number of weights, and it is that per-sweep cost, not the unit count, that dominates: since the number of weights is typically much larger than the number of units, the overall computation ends up scaling like the square of the number of weights, not linearly with either weights or units. The total cost of the exact Hessian is polynomial in the size of the network, but the polynomial is the weight count squared, not a simple multiple of a single gradient's cost by the number of units. That distinction is the one this book keeps testing for. A polynomial blowup — quadratic effort in the weight count instead of linear — is a cost that a faster machine erases; it is exactly the kind of fact a 2026 workstation renders moot for any network small enough that anyone in 1992 would have trained it. Hecht-Nielsen's own book preserves a vivid case of that hardware ceiling in the act of being hit: when McInerney, Haines, Biafore, and Hecht-Nielsen went looking for a genuine local minimum of a backpropagation error surface — as opposed to the shallow plateaus every practitioner complained about — "it was somewhat surprising" when they actually found one, because "finding this local minimum was not easy." It required using "the Symbolics Corporation's MacsymaTM mathematics expert system running on the largest Digital Equipment Corporation VAXTM computer to generate approximately a megabyte of FORTRAN code" encoding the closed-form error function and its first and second partial derivatives, and then "a 12-hour run on a Cray-2" to search the resulting surface — after which the discovery was confirmed by exactly the kind of calculation this section has just derived: showing that the gradient vanished and that "the Hessian of this function...was strongly positive-definite at this point." A 2026 laptop repeats that Hessian calculation, for a network of the size they were studying, before the kettle boils; nothing about the mathematics of Bishop's exact algorithm changes, only the wait. Set beside that dissolving fact is the one the previous section already drew from Blum and Rivest and that no amount of forward-and-backward sweeping touches: whether weights exist at all that fit a training set exactly is, for a two-unit hidden layer, NP-complete. Backpropagation's gradient, first-order or second-order, never asks that question — it only ever asks which direction is downhill from here, cheaply, and that is precisely why it could be run at all on the hardware this section has just watched groan under it.
The rerun this section owes the McInerney, Haines, Biafore, and Hecht-Nielsen anecdote from the previous section is not a repetition of their exact experiment — no surviving text states the size or task of the network whose error surface they searched, only the machinery it took to search it. What does survive is the reasoning that made the search necessary in the first place, and that reasoning can be rerun directly. Hecht-Nielsen records the received wisdom before their 1988 discovery: "many backpropagation error surfaces are dominated by flat areas and troughs that have very little slope," so shallow that "the generalized delta rule has a hard time determining which way to move the weight to reduce the error," and "great numerical precision (e.g., 32-bit floating point) and patience must be employed to make significant progress." The standing assumption was that these flat stretches were never genuine minima: "experience suggested that such minima might not exist because usually when training failed to make downhill progress and the error level was high it was discovered that further patience ... would eventually lead to the weight moving away from what was clearly a shallow spot on the surface and onto a steeper part." What follows reruns exactly that test — patience, then an exact Hessian — on the textbook network the previous two sections built by hand: the two-hidden-unit sigmoid solution to exclusive-or.
The protocol runs the exact equations of Section 2, not a paraphrase of them. Two inputs, two hidden sigmoid units, one sigmoid output unit — nine free parameters once each unit's bias is folded in as an extra weight from a fixed input of 1 — trained by full-batch gradient descent on the sum-squared error E = (1/2)Σ_k(t_k − o_k)² over the four exclusive-or patterns, with the δ_k and δ_j recursion of Section 2 computing every derivative. Four hundred networks were initialized independently, weights drawn from a zero-mean Gaussian, and run for 4,000 steps of gradient descent each, all four hundred computed together as one batch of array operations rather than four hundred separate loops. Wall-clock time for the whole search: 0.345 seconds. Contrast that number, not the mathematics, with McInerney and colleagues' twelve hours on a Cray-2: the difference is a fact about 1988 supercomputers and 2020s laptops, not about backpropagation. Most of the four hundred runs did what four decades of practice says they should — settled toward one of exclusive-or's many zero-error global minima, weight permutations of each other, exactly as Hecht-Nielsen's discussion of weight-permutation symmetry predicts. One did not. Run 96 stalled at a sum-squared error of 0.499997, with a gradient norm of 1.1×10⁻⁴ — a number small enough that a practitioner watching only the gradient would have declared the run converged, and stuck.
Apply the "further patience" test first, since it cost McInerney's contemporaries nothing but time and it costs a modern machine less. Run 96's stalled weights were used to seed 200,000 further steps of the identical gradient descent, at a smaller step size — trivial extra compute on 2020s hardware, the kind of patience nobody in 1988 could spend on a single run without a supercomputer allocation. The stall broke. Error fell from 0.499997 to 6.3×10⁻⁵, and the gradient norm fell in step, from 1.1×10⁻⁴ to 4.1×10⁻⁵. Measured this way, the rerun reproduces Hecht-Nielsen's reported experience exactly: the shallow spot was not a trap, it just needed more steps than a 1988 budget made comfortable, and on hardware where steps are free the distinction between "stuck" and "slow" evaporates. That would be a tidy place to stop, except that it begs the question the exact Hessian exists to answer without waiting: how would anyone have known, from the gradient alone, that run 96 was merely slow rather than stuck at a genuine local minimum, before spending the extra 200,000 steps to find out?
Answering that question without waiting is exactly what McInerney and colleagues' Hessian was for, and exactly why they needed Macsyma and a Cray-2 to get one: their claimed discovery "was proven by showing that all of the first partial derivatives of the mean squared error function went to zero at this point and that the Hessian of this function (the matrix of second partial derivatives) was strongly positive-definite at this point" — a vanishing gradient alone proves nothing, since a saddle point and a flat plateau both vanish there too; only a positive-definite Hessian, every eigenvalue strictly greater than zero, certifies that every direction out of the point is uphill. Sympy stands in here for Macsyma, and a laptop for the Cray-2 and its megabyte of FORTRAN: differentiating the same nine-parameter closed-form error function symbolically, twice, to build the exact 9×9 Hessian took 0.44 seconds, and evaluating both the gradient and the Hessian at any given point in weight space took roughly two milliseconds after that. Evaluated at run 96's stalled point — the one whose gradient norm alone looked like convergence — the exact Hessian's eigenvalues came out as [−1.95×10⁻³, −2.61×10⁻⁴, ≈0, 2×10⁻⁶, 3.7×10⁻⁵, 1.27×10⁻⁴, 1.75×10⁻³, 6.44×10⁻³, 0.500]: two negative eigenvalues, unambiguously not positive-definite. Run 96 was never a local minimum, only a saddle so flat that the gradient could not tell the difference at the tolerance a practitioner would ordinarily accept — which is exactly what the extra 200,000 steps then went on to demonstrate empirically, at the cost of a coffee break rather than a supercomputer allocation. Evaluated instead at the point those extra steps converged to, the eigenvalues came out as nine positive numbers, from 3.6×10⁻⁶ up to 4.6×10⁻⁴: a genuine, if very shallow, local minimum, certified in milliseconds by the same closed-form Hessian that took McInerney's team a dedicated expert system and twelve hours of supercomputer time to compute for their own, larger network.
Section 2 also reported a specific asymptotic claim from Bishop's exact-Hessian algorithm: that the cost of the full second-derivative matrix scales with the square of the number of weights, not linearly with the unit count. That claim can be checked directly rather than taken on Bishop's accounting. Building networks of the same 2-H-1 exclusive-or shape with H ranging from 2 to 48 hidden units — 9 to 193 weights — and computing the full Hessian by finite-differencing the analytic gradient (an O(W) evaluation, differenced across all W directions, for an O(W²) total cost, deliberately the naive numerical route rather than Bishop's more efficient exact algorithm), the measured times were: 0.0011 s at 9 weights, 0.0032 s at 17, 0.0109 s at 33, 0.0403 s at 65, 0.1705 s at 129, and 0.3836 s at 193. The ratio of time to weight-count-squared holds nearly constant across that whole range — between 9.5×10⁻⁶ and 1.3×10⁻⁵ seconds per squared weight throughout — and a log–log regression of time against weight count over the six sizes returns a slope of 1.92, close to the exponent of 2 that Bishop's accounting predicts and far from the exponent of 1 that scaling with the unit count would have implied. The mathematics Bishop described is confirmed on this machine as it was on his; what has changed is only that a computation whose polynomial exponent once justified writing an entire paper about how to make it efficient — worth optimizing, in 1992, because a naive version was expensive enough to matter — now finishes for a 193-weight network in well under half a second even taken the naive way, un-optimized, exactly the substitution this book keeps making: the exponent survives, the wait does not.
Sort what this rerun found by the question this book keeps asking. The twelve-hour Cray-2 run, the megabyte of Macsyma-generated FORTRAN, the four hundred networks that in 1988 would have had to be trained one at a time rather than as a single batch of array arithmetic, the 200,000 extra steps of "patience" that only a free machine can spend without a supercomputer allocation — all of that is 1988 hardware, and none of it survives contact with a laptop that finishes the equivalent work in well under a second. But three things this rerun turned up are not hardware facts, and a 2026 datacenter does not make them go away. First, a vanishing gradient is not evidence of a minimum — Section 2's whole recursion computes a first derivative, and a first derivative alone cannot distinguish a saddle from a trough from a genuine floor, no matter how fast or how many times it is evaluated; only the sign pattern of the second derivative can, which is exactly why Bishop's algorithm and McInerney's Cray-2 run were both spent on the same matrix rather than a faster gradient. Second, that matrix's cost is provably quadratic in the number of weights — this rerun's own measurement put the exponent at 1.92, not the 1.0 that scaling with unit count would give — a polynomial fact about the algorithm, unmoved by which machine executes it, only made cheap enough that at these network sizes cheap and free are indistinguishable. And third, sitting behind both of those observations is Blum and Rivest's harder fact from Section 1, which no gradient or Hessian in this section ever touched: whether a fitting set of weights exists at all, for a network with even one hidden layer, is NP-complete, a claim about every future machine and not merely this one. Backpropagation's gradient and Bishop's Hessian both answer a softer question — which way is downhill, and is downhill really over — and this rerun is a demonstration that both of those softer questions are now nearly free to ask. The harder question underneath them was never on the table.
Chapters 1 and 2 treated the input to a network as an unstructured vector: the perceptron's x and the multilayer network's x carried no information about which components were spatial neighbors, and nothing in the discriminant function w·x or in backpropagation's chain rule cared whether component 37 sat next to component 38 on a retina or was drawn from an unrelated sensor across the room. A photograph is not that kind of vector. Shift a digit two pixels to the right, or bend a stroke slightly, and a fully-connected network sees a training example essentially unrelated to the original — every one of its weights was fitted to the old pixel positions — even though a human, and presumably an animal's visual system, calls it the same digit. Kunihiko Fukushima's neocognitron, published in 1982 as an explicit successor to his earlier cognitron, was built to confront exactly this gap: earlier pattern-recognition models, he wrote, shared the property that "the response of most of these models... was severely affected by the shift in position and by the distortion in shape of the stimulus patterns," so that "these conventional models did not have an ability to recognize position-shifted or deformed patterns." The neocognitron's architecture was a direct translation, into a layered network of modifiable synapses, of a physiological hypothesis about how the mammalian visual cortex avoids the same failure — Hubel and Wiesel's account of a hierarchy of increasingly selective, increasingly position-tolerant cells running from the retina to association cortex.
Fukushima's model borrows the hierarchy directly: its structure "somewhat resembles the... hierarchical model of the visual nervous system proposed by Hubel and Wiesel," which runs from the lateral geniculate body through simple cells, complex cells, and lower- and higher-order hypercomplex cells, with the property that "a cell in a higher stage generally has a tendency to respond selectively to a more complicated feature of the stimulus pattern, and, at the same time, has a larger receptive field, and is more insensitive to the shift in position." The neocognitron implements this with two alternating cell types per stage — S-cells, which behave like simple or lower-order hypercomplex cells and whose afferent synapses are the only ones subject to learning, and C-cells, which behave like complex or higher-order hypercomplex cells and whose synapses are fixed. Critically, the S-cells of a given stage are not each wired independently: "it is hypothesized that all the cells in a single cell-plane have afferent synapses of the same spatial distribution, and only the positions of the presynaptic cells are shifted in parallel depending on the position of the postsynaptic cells" — one learned filter, replicated at every position across a two-dimensional "cell-plane," so that a feature learned at one location is automatically detected at every other location without separate training. Cascading these modules has a geometric consequence that the derivation in the next section will make precise: "the deeper the layer is, the larger becomes the receptive field of each cell of that layer," until in the deepest module the receptive field of each C-cell becomes "so large as to cover the whole input layer" and the network's response no longer depends on where in the image the pattern appeared at all.
Yann LeCun and his co-authors at Bell Labs arrived at nearly the same architectural idea from a different direction: not biology, but the statistical-learning-theory concern that will occupy Volume II's later chapters — an over-parameterized network overfits. "The basic design principle is to reduce the number of free parameters in the network as much as possible without overly reducing its computational power," they wrote of the network trained on 16-by-16 pixel images of handwritten zip-code digits from U.S. mail, citing this as a route to "a reduced Vapnik-Chervonenkis dimensionality." Their mechanism for cutting parameters without cutting the pixels the network could see was the same replication Fukushima had used for a different reason: "the detection of a particular feature at any location on the input can be easily done using the 'weight sharing' technique... having several connections (links) controlled by a single parameter (weight)," so that "all units in a plane share the same set of weights, thereby detecting the same feature at different locations." Each unit in their first hidden layer, organized into twelve 8-by-8 "feature maps," looked at a 5-by-5 patch of the input and shared its 25 weights with all 63 other units in its map, "since the exact position of the feature is not important," so that "the function performed by a feature map can... be interpreted as a nonlinear subsampled convolution with a 5 by 5 kernel." The bookkeeping made the saving explicit: layer H1 alone connected 768 units with 19,968 individual synapses, but because they were tied together in groups of 25, the layer carried only 1,068 independently adjustable numbers. Fukushima had built shift invariance into a hand-designed hierarchy modeled on cortex; LeCun's group derived the identical replication rule from a wish to keep the hypothesis space small enough for backpropagation, applied to raw pixels rather than hand-built features, to generalize at all — the same architectural fix reached from opposite ends of the book's two founding traditions.
Both the neocognitron and LeCun's shared-weight network are supervised or self-organized-but-hand-structured architectures: the spatial regularity of images is wired in before any data is seen, and, in LeCun's case, corrected afterward by backpropagation against labeled targets. Teuvo Kohonen's self-organizing map, published in the same period, addressed a related but distinct question — not how to make a supervised classifier tolerant of spatial shift, but how topological structure could emerge, with no teacher and no labels at all, out of the statistics of the input alone. Kohonen's own framing reaches past neuroscience toward the oldest question in the connectionist tradition: "the possibility of constructing spatial maps for attributes and features in fact revives the old question of how symbolic representations for concepts could be formed automatically." His model dispenses with layers entirely in favor of a single array: "a one- or two-dimensional array of processing units which did not have this structure initially" is shown a sequence of input vectors, and "the principal discovery is that... the signal representations are automatically mapped onto a set of output responses in such a way that the responses acquire the same topological order as that of the primary events" — nearby inputs come, through training alone, to activate nearby units, even though the array started with no notion of which unit was near which. The mechanism, as the next section derives in full, needs only the ingredients Kohonen lists directly: an array of processing units which receive coherent inputs from an event space and form discriminant functions of those inputs; a mechanism which selects the unit whose discriminant function responds most strongly; some kind of local interaction that simultaneously activates the selected unit and its nearest neighbours; and an adaptive process that strengthens the activated units' response to the present input. Where Fukushima built position-tolerance into fixed wiring and LeCun's group tied weights to control capacity for a supervised gradient, Kohonen showed that topographic order — the same kind of order Hubel and Wiesel had found already present in cortex — could be a statistical consequence of competition and local cooperation, requiring no target signal to arrive from anywhere. The three papers in this setup section are, in that sense, three different answers to one question raised by moving from vectors to grids: how does a network come to know that some of its inputs are neighbors?
Fukushima and Miyake give the neocognitron as two closed-form cell equations, one per cell type, and the architecture the setup section described falls out of iterating them. Write ℓ for the module (stage) index, kℓ for the index of an S-plane or C-plane within that module, and n for the two-dimensional position of a cell within its plane; uCℓ(κ,n) is the output of a C-cell and uSℓ(kℓ,n) of an S-cell. An S-cell first forms an excitatory sum over all Kℓ-1 preceding C-planes and over an offset ν ranging across a fixed connection area Sℓ: e = Σκ Σν aℓ(κ,ν,kℓ)·uC,ℓ-1(κ,n+ν), where aℓ(κ,ν,kℓ) is a modifiable excitatory synapse shared by every cell in the plane. The cell's output is then the shunting-inhibition ratio the paper writes as Eq. (1), uSℓ(kℓ,n) = φ[ (1+e) / (1 + (rℓ/(1+rℓ))·bℓ(kℓ)·vC,ℓ-1(n)) − 1 ], with φ[x] = max(x,0), where bℓ(kℓ) is a modifiable inhibitory synapse, rℓ a fixed gain that "controls the intensity of the inhibition" — the larger it is, "the more selective becomes the cell's response to its specific feature" — and vC,ℓ-1(n) an inhibitory cell's output, the weighted root-mean-square vC,ℓ-1(n) = [Σκ Σν cℓ-1(ν)·uC,ℓ-1(κ,n+ν)²]^{1/2} (Eq. 2), with weights cℓ-1(ν) fixed, decreasing in |ν|, and normalized so Σν cℓ-1(ν)=1 (Eq. 3). Of these five symbols, only a and b are trainable; rℓ and cℓ-1 are wired in once, at construction, and never change. The C-cell, one stage later, pools the S-cells of the one S-plane that shares its feature index over a second fixed area Dℓ: uCℓ(kℓ,n) = ψ[ (1 + Σν dℓ(ν)·uSℓ(kℓ,n+ν)) / (1+vSℓ(n)) − 1 ] (Eq. 4), where ψ[x]=x/(a+x) for x≥0 and 0 otherwise, and vSℓ(n) is the arithmetic mean of the same S-cells (Eq. 5) — an unmodifiable, saturating pooling that is exactly why a shifted feature still fires the C-cell: whichever S-cell in the neighborhood N happens to see the feature now, the sum over ν picks it up. Neither Sℓ nor Dℓ is a free parameter of training; both are architectural choices, and Fukushima is explicit that they grow with depth: "the size of the connection area S£ of these cells is set to be small in the first module and to increase with the depth £," and, of the C-cell's pooling area, "the size of the connection area D£ of these cells is set to be small in the first module and to increase with the depth £." Because each S-cell's receptive field is the union, over the offsets in Sℓ, of C-cell receptive fields that were already unions over Dℓ-1 at the previous stage, the receptive field size composes multiplicatively across modules — the geometric growth the setup section deferred to here — until, as stated already, the deepest module's C-cells see the entire input layer and code only for pattern identity, not position.
Training touches only aℓ and bℓ, and it does so by a competitive rule that is worth separating from the weight-sharing already built into the equations, because the two are not the same mechanism. Weight sharing, as Section 1 showed, is guaranteed by fiat: every S-cell in a plane is defined to have "afferent synapses of the same spatial distribution." What self-organization has to decide is which feature each of the (initially near-random) S-planes should come to detect, and this is where competition enters. Within an S-layer, cells whose receptive fields sit at nearly the same input location are grouped into an "S-column," "closely resembl[ing] that of 'hypercolumns' proposed by Hubel and Wiesel." Every time a stimulus is presented, "the S-cell which is yielding the largest output is chosen as a candidate for the representatives" from each S-column, and if a plane produces more than one candidate across columns, "only the one which is yielding the largest output among them is chosen as the representative from that S-plane" — at most one winner per plane, per presentation. Only the winner's active synapses are changed, by Δaℓ(κ,ν,kℓ) = qℓ·cℓ-1(ν)·uC,ℓ-1(κ,n+ν) (Eq. 6), Δbℓ(kℓ) = qℓ·vC,ℓ-1(n) (Eq. 7), where qℓ is a fixed learning-rate constant — a Hebbian-shaped update, proportional to the presynaptic signal actually present, that only ever increases synaptic strength (there is no decay term). The trick that keeps this compatible with the shared-weight assumption is procedural rather than architectural: "all the other S-cells in the S-plane, from which the representative is chosen, have their afferent synapses reinforced by the same amounts as those for their representative" — every non-winning cell in the winning plane receives the identical Δa, Δb computed at its own (shifted) receptive field, so that if the plane started identical across positions it stays identical, without any single stored parameter being literally shared the way LeCun's group implemented it three years later. Planes that produced no representative at all that round are untouched: "the cells in the S-planes, from which no representative is chosen... do not have their afferent synapses modified at all." The result is winner-take-all competition among planes coupled with parallel, position-invariant updates within the winning plane — a self-organizing route to exactly the property, "detecting the same feature at different locations," that the setup section's supervised network reached instead by tying one parameter to 25 connections and correcting it with a label-driven gradient.
Kohonen's derivation needs no cell types at all, only a single array of identical units and three equations. Let x=[ξ1,ξ2,...,ξn]^T ∈ Rⁿ be the input vector at time step t, presented to every unit of the array simultaneously (the "relaying network" of Fig. 1 is assumed transparent). Unit i holds an input weight vector mi=[μi1,μi2,...,μin]^T ∈ Rⁿ and forms the discriminant function Kohonen writes as Eq. (1), ηi = Σj ξj μij = mi^T x. A discrimination mechanism then selects the single unit of maximum response, Eq. (2): ηk = max{ηi}. This one step already does the work weight-sharing did for Fukushima and LeCun: instead of wiring in tolerance to shift by replicating a filter across positions, Kohonen replicates nothing — every unit's mi is independent — and instead forces a competition in which, at any instant, exactly one unit is selected, whatever the input. Learning then touches only the winner k and a fixed geometric neighborhood around it: "for unit k and all the eight of its nearest neighbours" (except at the array's edges, where fewer exist) an adaptive process is assumed active, Eq. (3), mi(t+1) = [mi(t) + α x(t)] / ‖mi(t) + α x(t)‖, applied for every i in that neighborhood, where α is a scalar "gain parameter" and the denominator renormalizes the result to unit length. Kohonen calls the rule close to, but pointedly not, the perceptron's: it resembles "the well-known teaching rule of the Perceptron, except that the direction of the corrections is always the same as that of x (no decision process or supervision is involved), and the weight vectors are normalized." There is no label anywhere in Eq. (3) — x supplies both the input and the only signal the update ever sees, and normalization means the update is a pure rotation: "the process of Eq. (3) does not change the length of m i but only rotates m i towards x." For the network to settle rather than oscillate, α must shrink across training — Kohonen made it "a function of the iteration step, e.g., proportional to lit" (i.e. 1/t), calling "a decreasing sequence... necessary for stabilization."
The two derivations solve different problems with a structural family resemblance: both replace supervision with competition, and both make the competition local rather than global in its consequences. Fukushima's competition is local within a plane — only one S-cell per S-column, and at most one representative per plane, is updated on a given presentation — but the resulting invariance is guaranteed globally and permanently, because every non-winning cell in the winning plane is forced to copy the winner's update at its own receptive field, so the plane is identical everywhere by construction from the first training step onward. Kohonen's competition is local within the array — one winning unit and its eight geometric neighbours are updated, everyone else left alone — but nothing forces any two units to end up alike; topological order, if it appears at all, is an emergent statistical property of many such local updates compounding over "a number of iteration steps," not a guarantee written into the update equation. That is why Kohonen needed simulations at all, where Fukushima's shift-invariance follows from Eq. (6)-(7) by inspection: the neocognitron cannot fail to be position-tolerant once built, but the self-organizing map can fail to order itself, and Kohonen says so directly, reporting that "several types of computer simulations are used to demonstrate the ordering process as well as the conditions under which it fails." One such condition he flags explicitly is symmetry-breaking: with a symmetric input distribution and no privileged direction built into the array, "there are eight equally probable symmetrical alternatives in which the map may be realized," so that which orientation it ultimately takes is arbitrary unless broken by a "seed" unit with fixed weights or by an asymmetric input distribution — a limitation of the mathematics of competitive dynamics under symmetry, not of any machine it ran on. Kohonen's own generalization confirms the two models are aimed at different targets: the array need not be shown geometric images at all. In Simulation 4, a one-dimensional chain of units received five inputs each, drawn at random from twenty mistuned resonators, "so that there is no initial correlation or order in any structure or parameters," and after a few thousand presentations of random sinusoids the chain organized into an ascending or descending frequency map — Table 1 lists the tuning each unit acquired, from 0.55 up to 0.98 in one run — solely from the statistics of "non-identical but coherent" inputs, with no spatial layout in the data to recover. This is the sense in which the two 1980s architectures answer the question the setup section posed from opposite directions: Fukushima wires a known geometric invariance into a supervised hierarchy so it need never be learned; Kohonen shows that an unsupervised competitive-cooperative rule can discover an a priori unknown order — geometric, tonotopic, or, by extension, purely abstract — from exposure to structured data alone. Neither result depended on scale: Kohonen's array, resonator bank, and iteration counts (thousands, not millions) ran to convergence on the modest computers available in 1982, and rerunning them today would change nothing about whether the ordering theorem holds or where it fails — the eightfold symmetry, the need for a decaying gain, the requirement that inputs be "coherent," are properties of the update equations, and a 2026 datacentre inherits every one of them unchanged.
Section 1 and 2 made three claims that are checkable rather than merely quotable: that tying one 5-by-5 filter to every position in a plane cuts the number of independently adjustable numbers by an order of magnitude without destroying accuracy; that the resulting network is more tolerant of a shifted input than a fully-connected one, though only a cascaded, multi-stage architecture like the full neocognitron is tolerant of shift by construction; and that Kohonen's competitive-cooperative update turns a randomly initialized array into a topologically ordered one, with an orientation that is arbitrary unless something breaks the symmetry. All three are reproduced below on 2020s hardware, in every case substituting modern, larger, more convenient data for what the original authors actually used. LeCun and his co-authors trained on 7291 16-by-16 pixel U.S. Postal Service zip-code digits and tested on 2007 more; this rerun uses a 4,000/1,500 split drawn from the 70,000-image, 28-by-28 pixel MNIST set, a later and unrelated corpus assembled for a different competition, because it is what sits in the pre-fetched cache and nothing about the weight-sharing argument depends on the digits' provenance. The rerun's "shared-weight" network is a single convolutional stage — one bank of learned 5-by-5 filters, average pooling, one linear classifier — trained by hand-derived backpropagation in plain numpy, not the two-hidden-layer H1/H2 cascade of the original paper and not the SN simulator LeCun's group used on a Sun-4/260. Kohonen's simulations are reproduced with a fifty-unit one-dimensional chain exposed to points drawn from a two-dimensional unit circle, the closest continuous analogue to his own experiments with two- and three-dimensional input distributions, run to convergence in under a second. Where the rerun disagrees with the original numbers, both are reported; where it agrees, that agreement is the finding, because it is evidence that what the papers described was a property of the update equations, not of the specific machine or dataset in front of them.
LeCun's group reported, after training their full two-layer weight-shared network for "23 passes through the training set (167,693 pattern presentations)," that "the percentage of misclassified patterns was 0.14% on the training set (10 mistakes) and 5.0% on the test set (102 mistakes)" — 95.0% test accuracy on 2007 held-out zip-code digits — adding dryly that "our training times were 'only' 3 days on a Sun workstation." The rerun's much smaller single-stage version — 12 shared 5-by-5 filters (312 parameters total, versus 19,968 individual synapses tied down to 1,068 independent numbers in LeCun's H1 alone), 2-by-2 average pooling, one linear classifier (4,330 parameters) — trained on the 4,000-image MNIST subsample in 25 epochs and under three seconds, reaching 97.15% training accuracy and 94.80% test accuracy: a total of 4,642 free parameters against a same-architecture fully-connected comparison network with 12 hidden units seeing the whole 28-by-28 image, which needed 9,420 parameters in its first layer alone and reached only 88.40% test accuracy under the same optimizer budget. A wider, more conventional fully-connected network (100 hidden units, ReLU, Adam) closed most of the gap — 92.93% — but at roughly seventeen times the shared-weight network's parameter count (79,510 versus 4,642). The direction of every one of LeCun's comparisons survives the substitution of MNIST for zip codes and of a laptop for a Sun-4/260: tying weights across a plane buys both fewer parameters and higher accuracy than the fully-connected alternative, at a wall-clock cost that fell from three days to three seconds.
Neither paper claims that weight sharing alone makes a network indifferent to translation; Section 1 was careful that full position-tolerance in the neocognitron is a property of the whole cascade, achieved only once "the deepest module's C-cells see the entire input layer," and LeCun's group built two pooling stages toward the same end. A single convolutional stage, tested directly, shows exactly this partial, not total, tolerance. Shifting every test image by two pixels along one axis and re-scoring both trained networks without retraining: the fully-connected network's accuracy fell from 88.40% to 47.27% (a horizontal shift) or 56.47% (vertical) — drops of 41 and 32 points — while the one-stage shared-weight network fell from 94.80% to 72.33% and 75.67% respectively, drops of 22 and 19 points. On a diagonal two-pixel shift in both directions at once, both networks degraded further (fully-connected to 29.4% and 29.8%; shared-weight to 49.1% and 44.1%), the shared-weight network's advantage narrowing as the shift compounds. The finding is quantitative confirmation of the qualitative claim the setup section drew from the two architectures: weight sharing and local pooling buy real, measurable shift tolerance — roughly half the accuracy drop of an equivalent fully-connected network under a two-pixel translation — but a single stage, lacking the second convolution-and-pooling round that both the neocognitron and LeCun's actual H1/H2 cascade provide, is not shift-invariant in the strong sense either paper claims for their full architecture. That gap between one stage and several is architectural, not computational; no amount of 2026 hardware closes it without adding the stages back.
Kohonen wrote that "several types of computer simulations are used to demonstrate the ordering process as well as the conditions under which it fails," and offered no closed-form guarantee of ordering the way the neocognitron's shift-tolerance follows from its equations by inspection — so this claim has to be rerun, not just re-derived. A chain of fifty units, each holding a random unit vector in the plane (order-metric, the correlation between a unit's index along the chain and the angular position of its weight vector, starting at −0.18 — indistinguishable from disorder), was exposed to 2,000 points drawn uniformly from the unit circle and updated by exactly Eqs. (1)-(3): winner selection by maximum inner product, update of the winner and its shrinking neighborhood, renormalization to unit length. After only 50 presentations the order-metric reached −0.993; after 2,000 it was −0.999, and the chain's fifty weight vectors had spread across 5.85 of the 6.28 radians in a full circle in strictly monotonic angular order — a smooth, one-dimensional embedding of the two-dimensional input manifold, recovered from an initial condition with, in Kohonen's words describing his own random restarts, "no initial correlation or order in any structure or parameters." The rerun reproduces the ordering itself in under a second of computation, on an input distribution chosen only because it is the simplest continuous space with the rotational symmetry Kohonen's own examples exploited.
Kohonen's remark that with a symmetric input distribution "there are eight equally probable symmetrical alternatives in which the map may be realized" is also directly testable: the same update rule, run from five different random seeds with nothing else changed, should order the chain but not agree on which way round it orders it. It does not agree. Three of five seeds produced an order-metric near +0.999 (angle increasing along the chain) and two produced −0.999 (angle decreasing); the chain's starting unit landed at angles 2.70, 1.34, −1.75, 1.67, and −2.93 radians across the five runs, scattered around the full circle with no repeated value. Every run reached the same quality of order — near-perfect monotonic correlation between position and angle — but which of the (continuum version of the) "eight equally probable symmetrical alternatives" it settled into was a property of which random input sequence it happened to see first, exactly as Kohonen's own account predicts and exactly as his proposed fix would be needed to control: he suggests breaking the symmetry either by defining "'seeds', i.e. units with fixed, predetermined input weights," or by using "nonsymmetrical distributions and arrays which might have the same effect." This is not an artifact that a bigger chain, more presentations, or faster hardware would remove: symmetry-breaking under a symmetric input distribution is a structural fact about the update rule, reproduced identically on a 2020s laptop as on Kohonen's own machine in 1982.
Every number in this section changed under the substitution of dataset and hardware, and every claim it was checking did not. The zip-code error rate LeCun's group reported (5.0%) is not the error rate this rerun's much smaller network reports on MNIST (5.2% — an agreement close enough to be almost suspicious, given how little the two architectures share beyond the weight-sharing principle), and the "3 days on a Sun workstation" it took to reach that number in 1989 became three seconds in this rerun; but the ranking between weight-shared and fully-connected networks, in both parameter count and accuracy, held at both scales, and the partial (not total) shift-tolerance of a single convolutional stage held at both scales too. Kohonen's array took "a number of iteration steps" measured in the low thousands to order itself in 1982 and takes a fraction of a second now, on an input distribution he never tried; the ordering happened anyway, and so did the failure mode he flagged — arbitrary orientation under a symmetric input, unresolved by more computation because it is not a computational shortfall. That is the pattern this book keeps finding on the far side of the GPU line: where a paper's number was a wall-clock report, the wall clock moved by five orders of magnitude and the finding survived; where a paper's claim was about what an equation does — guarantees position-tolerance by construction, or fails to guarantee an orientation under symmetry — the claim was unmoved by any of it, because it was never about the machine in the first place.
A conventional computer memory is addressed: to retrieve a word, the processor supplies its location and the memory returns whatever bit pattern is stored there, with no obligation that the pattern be related to the address. John Hopfield's 1982 paper "Neural Networks and Physical Systems with Emergent Collective Computational Abilities" (Proceedings of the National Academy of Sciences 79) proposed a different kind of memory, addressed by content: supply a fragment or a noisy version of a stored pattern, and the system should settle onto the complete, clean pattern that most resembles it. Hopfield's device for building such a memory was not an algorithm in the usual sense but a physical analogy. He treated a network of binary neurons, symmetrically interconnected, as a dynamical system with a scalar "energy" that could only decrease as the system evolved — the same mathematical object, formally, as the Hamiltonian of an Ising spin glass. Stored memories were arranged to sit at the bottom of local wells in this energy surface; recalling a memory meant releasing the system near a well and letting the dynamics roll it downhill. The 1982 paper itself is not among the primary sources available here, but Hopfield restated its model and its Content-Addressable-Memory (CAM) construction directly, in his own notation, in a 1985 paper with David Tank on optimization; and Ackley, Hinton, and Sejnowski's 1985 paper on the Boltzmann machine restates the same energy-minimization argument from outside, citing Hopfield's convergence result as the fact their own stochastic generalization would have to relax. This section draws on both restatements to set up the energy-landscape formalism that the rest of the chapter — first the Boltzmann machine, then the Helmholtz machine — will modify piece by piece.
Hopfield and Tank set out the network as a circuit of "neurons" — amplifiers with sigmoid input-output relations $V_j = g_j(u_j)$, where $u_j$ is the input voltage and $V_j$ the output — connected through a matrix of conductances $T_{ij}$, with an external bias current $I_i$ into each unit. Dividing through by the membrane capacitance, the equation of motion for the whole network is written $$\frac{du_i}{dt} = -\frac{u_i}{\tau} + \sum_{j=1}^N T_{ij}V_j + I_i, \qquad V_i = g(u_i),$$ matching their equation (3), with $\tau = RC$ the single time constant shared by all units. The paper then states the key fact carried over from the discrete 1982 model: "the stable states of a network comprised of N neurons are the local minima of the quantity" — their equation (4), which in cleaned-up notation (their OCR-mangled subscripts silently restored) reads $$E = -\tfrac{1}{2}\sum_{i=1}^N\sum_{j=1}^N T_{ij}V_iV_j - \sum_{i=1}^N V_iI_i.$$ Here $V_i \in \{0,1\}$ (or, in the original 1982 formulation Hopfield restates elsewhere in the same paper, $\{-1,+1\}$) is the state of neuron $i$, and the sum runs over all pairs. Because the connections are symmetric, $T_{ij}=T_{ji}$, this $E$ is a Lyapunov function for the dynamics: "The state space over which the circuit operates is the interior of the N-dimensional hypercube defined by" $V_i = 0$ or $1$, and in the high-gain limit — each amplifier driven toward saturation — the minima of $E$ collapse onto the $2^N$ corners of that hypercube. A stable configuration of the network is, by construction, a local minimum of $E$ among all single-neuron flips.
To turn this energy-minimizing circuit into a memory, Hopfield and Tank work the construction backward: choose the $T_{ij}$ so that a chosen set of patterns are themselves the local minima of $E$. They start from the observation that if the desired stable states are a set of $m$ binary vectors $V^s$, $s=1,\dots,m$, a natural candidate energy is quadratic in the overlap between the current state and each stored pattern, $$E = -\tfrac{1}{2}\sum_{s=1}^m (V\cdot V^s)^2.$$ Matching this quadratic form to the standard form of Eq. (4) fixes the weights as an outer-product, or Hebbian, sum over the stored patterns, $T_{ij} = \sum_{s=1}^m V_i^sV_j^s$ — "the storage algorithm presented earlier (Hopfield, 1984) except for an additive constant." Recall is then dynamics, not lookup: "A memory, stored in the network by an appropriate choice of Tq elements, could be" retrieved "by setting the outputs of the amplifiers in the binary pattern of the recall key and then allowing the system to converge to a stable state," a state that "was interpreted as the memory word evoked by the key." Put in the vocabulary of the energy landscape: the recall key is a starting point somewhere on the hypercube, gradient descent on $E$ carries it downhill, and it comes to rest in whichever well is nearest — a solved instance of deciding "which of the stored binary words was" "closest" to the key word. A partial or corrupted cue is recovered exactly when it lies within the well's basin of attraction. The 1982 paper and the later statistical-mechanics analyses of the model (Amit, Gutfreund, and Sompolinsky's among them) go on to show that patterns stored too densely, or too correlated with one another, merge wells or spawn spurious minima — stable states of the network that are not memories anyone stored. That result is well established in the literature on this model, but it is not something Hopfield and Tank's 1985 paper itself demonstrates or even mentions, and none of the papers making that argument are among the primary sources drawn on here; it is flagged rather than derived. The motivation this chapter actually documents for moving to a stochastic rule is the one Ackley, Hinton, and Sejnowski give directly, taken up in the next block: not spurious minima but the asymmetry between settling into whichever well is nearest and sampling from a full distribution over wells.
Ackley, Hinton, and Sejnowski open their own paper by restating exactly this machinery from outside, in a slightly different notation, before turning it to a different purpose. Their network is a set of binary units $s_i \in \{0,1\}$, each representing whether "the system currently accepts or rejects some ele- mental hypothesis about the domain," joined by symmetric weights $w_{ij}$ and thresholds $B_i$; "the energy of a global configuration is defined as" $E = -\sum_{i<j}w_{ij}s_is_j + \sum_iB_is_i$, their equation (1) — the same bilinear form as Hopfield and Tank's Eq. (4), with connection strength standing in for conductance and threshold for bias current. And they state the convergence guarantee that Hopfield's 1982 paper had supplied and that their own stochastic algorithm would have to generalize: "A simple algorithm for finding a combination of truth values that is a local minimum is to switch each hypothesis into whichever of its two states yields the lower total energy given the current states of the other hypotheses. If hardware units make their decisions asynchronously, and if transmission times are negligible, then the system always settles into a local energy" minimum, citing Hopfield (1982) directly for the result, "Because the connections are symmetric" — the same symmetry Hopfield and Tank invoke for their continuous case. Asynchronous, one-unit-at-a-time updating is essential to the proof: flip more than one unit at once and the energy can rise, because $E$ is only guaranteed non-increasing along single-coordinate moves. This is the entire content of "content-addressable memory as an energy landscape": a hand-built quadratic surface over the corners of a hypercube, a Hebbian rule for carving stored patterns into its wells, and a local, asynchronous, gradient-following dynamics that is provably attracted to those wells and no others. The next section keeps the surface and the symmetry but replaces the deterministic downhill rule with a stochastic one — the move from settling into a minimum to sampling from a Boltzmann distribution over all of them.
Hopfield's construction gave a network a way to settle, deterministically and asynchronously, into whichever energy well was nearest. Ackley, Hinton, and Sejnowski's 1985 Boltzmann machine kept the same bilinear energy — the identical quadratic form set out in the previous section, $E = -\sum_{i<j}w_{ij}s_is_j + \sum_iB_is_i$ — but replaced the "always downhill" update rule with a stochastic one: a unit turns on or off as a probabilistic function of its neighbors' states and the weights on its links, so that at a fixed "temperature" $T$ the network no longer settles into one local minimum but samples repeatedly among the low-energy configurations, spending time in each in proportion to a Boltzmann distribution over global states. Simulated annealing — starting at high temperature, where the network wanders freely over the whole hypercube, and cooling slowly toward the operating temperature — is the mechanism by which the network is driven toward this equilibrium distribution rather than trapped in whichever well happens to be nearest the start. This changes what the network is for. A Hopfield net stores patterns as wells; a Boltzmann machine is trained so that its own equilibrium probability of a visible configuration matches the probability of that configuration in the training environment. The authors state the goal directly: the network's connections are adjusted "so as to construct an internal generarive model that produces examples with the same probability distribution as the examples it is shown." Recall becomes inference rather than descent: the network can "interpret" an example "by finding values of the variables in the internal model that would generate the exam- ple," and "when shown a partial example, the network can complete it by finding internal variable values that generate the partial example and using them to generate the remainder." Hidden units, which had no clear role in Hopfield's construction beyond providing extra storage capacity, now have a specific job: representing whatever latent structure the visible data does not directly reveal.
Ackley, Hinton, and Sejnowski's learning rule follows from asking a specific question: given the network's equilibrium probability distribution over its visible units, $P'(V_\alpha)$, and the distribution $P(V_\alpha)$ that the environment actually imposes when it clamps examples onto those units, how should the weights change to make $P'$ closer to $P$? They measure the discrepancy between the two distributions with a quantity they call $G$ — an asymmetric divergence of exactly the kind Kullback (1959) had defined — and differentiate it with respect to a single weight $w_{ij}$. Because the network's dynamics is governed by the same energy function in both the clamped and the free-running case, the algebra collapses to a strikingly simple result: $\partial G/\partial w_{ij}$ is proportional to $p_{ij} - p'_{ij}$, where $p_{ij}$ is the probability that units $i$ and $j$ are both on at equilibrium when a training example is clamped on the visible units, and $p'_{ij}$ is the probability that they are both on at equilibrium when the network is left completely free-running, generating from its own model with nothing clamped. Both quantities are estimated the same way — by annealing the network to equilibrium and counting co-occurrences — which is what makes the rule usable at all: it needs no information about a unit's role or position in the network, only a pairwise statistic of exactly the kind each unit and its neighbor can observe locally. The paper's own description of the two measurement phases is procedural rather than symbolic: in the first, "each environmental vector in turn was clamped over the visible units" and, once the network settled, "statistics about how often pairs of units were both on together were gathered at equilibrium" — this is $p_{ij}$, the positive phase. In the second, "the network was completely unclamped and allowed to reach equilibrium at a temperature of 10," and the same co-occurrence statistics were collected with nothing clamped — this is $p'_{ij}$, the negative phase. The weight update is then the sign of the difference between the two: "all weights in the network were incremented or decremented by a fixed weight-step of 2," with the sign of the step set by the sign of $p_{ij}-p'_{ij}$. When the two phases agree — when the network's unconstrained fantasies already reproduce the co-occurrence statistics the environment imposes — the gradient vanishes and learning stops, precisely at the point where $P'=P$.
Both $p_{ij}$ and $p'_{ij}$ are equilibrium statistics, and reaching equilibrium is exactly the operation Ackley, Hinton, and Sejnowski's own construction rules out cheaply: the network must be annealed from a high temperature down to the operating temperature every single time either phase is measured, for every one of the thousands of weight updates a run requires. Their own small demonstration makes the cost concrete. To train a 4-2-4 encoder — four visible units in each of two groups, feeding through a bottleneck of two hidden units — every measurement of $p_{ij}$ or $p'_{ij}$ required an annealing schedule of roughly two time-units at temperature 20, two more at 15, two more at 12, and four at 10, before ten further time-units of statistics-gathering at the operating temperature — where one time-unit was itself defined as enough random single-unit probes for every unit in the network to be given, on average, one chance to change state. That whole annealing-and-measuring cycle had to run twice per weight update — once clamped, once free — and "in 250 different tests of the 4-2-4 encoder, it always found one of the global minima," but the paper reports a median of roughly 110 such learning cycles to succeed and a longest run of roughly 1,810 (the source text's OCR renders these as "I10" and "18 10," unmistakably 110 and 1810). For a network with a four-bit visible layer and two hidden units, that is already thousands of full annealing schedules. This is the cost the authors flag themselves in their introduction — "the current version of the learning algorithm is very slow" — and it is worth asking whether it is a 1985 hardware ceiling or something that survives a 2026 datacenter. It is the latter, at least in kind: annealing a stochastic network to thermal equilibrium is a Markov chain Monte Carlo computation, and its cost is measured in mixing time — the number of steps the chain needs before its distribution is close to stationary — which is a property of the energy landscape's structure, not of the clock speed evaluating each step. Faster hardware runs each of the required steps faster, but for chains whose mixing time scales badly with the number of units (as it generically does whenever the landscape has many separated wells, exactly the regime the Hopfield construction was built to create), no constant-factor speedup removes the scaling problem itself. What Ackley, Hinton, and Sejnowski's slowness actually forced onto the field was the question the next section takes up: whether a network could be trained to model a data distribution without paying, on every single weight update, for a full equilibration of the negative phase.
Dayan, Hinton, Neal, and Zemel's 1995 Helmholtz machine answers that question by giving up on the equilibrium distribution as the object to be matched, and giving up on a single network doing both jobs. Where the Boltzmann machine used one set of symmetric weights for both settling and generating, the Helmholtz machine splits the work: "a recognition model is used to infer a probability distribution over the underlying causes from the sensory input, and a separate generative model, which is also learned, is used to train the recognition model." The generative model, with parameters $\theta$, defines a distribution over "explanations" $\alpha$ — complete assignments of activity to every hidden unit — and over how each explanation generates the visible data $d$; the log probability of the data, $\log p(d\mid\theta)$, requires summing over all such explanations, which the paper notes is exactly the sum that makes standard maximum-likelihood approaches such as the Expectation-Maximization algorithm intractable once the explanations number in the exponentially many. The move that avoids the sum is the same move underlying every variational bound that follows it in this book's later chapters: instead of averaging over the true posterior distribution over explanations, $P$, which is what an exact calculation would require, "we use a more convenient probability distribution, $Q$" — produced by the recognition network's own bottom-up weights $\phi$, and constrained to be factorial (independent unit by unit within a layer) so that it costs no more than a single feedforward pass to evaluate. Substituting $Q$ for $P$ does not, in general, recover the true log-likelihood, but it does not need to: the gap between the two is a Kullback-Leibler divergence, and "this term cannot be negative, so by ignoring it we get a lower bound on the log probability of the data given the model." Maximizing that lower bound — the negative of a quantity the authors call the (Helmholtz) free energy $\mathcal{F}$ — with respect to both the generative parameters $\theta$ and the recognition parameters $\phi$ simultaneously is the entire training procedure: no equilibration, no annealing schedule, just two feedforward computations (recognition up, generation down) and a bound that is cheap to evaluate because $Q$ was chosen to be cheap to evaluate.
Computing the exact gradient of $\mathcal{F}$ for the deterministic Helmholtz machine is still complicated, because a change in one layer's recognition weights propagates its effect through every layer above. The paper's answer — borrowed explicitly from the Boltzmann machine's own two-phase design and named the wake-sleep algorithm — keeps the split between generative and recognition weights but trains each with the other held fixed, one phase at a time. In the wake phase, data from the world are presented at the input layer and propagated upward, with each layer's units set stochastically according to the recognition weights; the top-down generative weights are then adjusted to make the layer below more probable under the generative model, given the sampled activity above — the same local delta rule Widrow and Stearns had described for supervised learning, now applied with the recognition network's own sampled states standing in for a teacher's labels. In the sleep phase the recognition weights are switched off entirely and the network runs the other direction: starting from the top layer, the generative weights alone produce a random top-down fantasy, layer by layer, exactly as the Boltzmann machine's free-running negative phase produced its own fantasy — but here in a single feedforward-in-reverse pass rather than a Gibbs-sampled equilibrium. "Since it has generated the instance, it knows the true underlying causes, and therefore has available the target values for the hidden units that are required to train the bottom-up weights," so the recognition weights can be updated by the same delta rule, this time treating the generative model's own sampled activity as ground truth. The paper is direct about the price of this shortcut: when the bottom-up and top-down activation functions are both the ordinary sigmoid, both phases reduce to exactly the same learning rule — what the authors call "the purely local delta rule" — which is precisely what makes the algorithm cheap — no annealing schedule, no repeated equilibration, one sample drawn per phase per weight update. But nothing in the construction guarantees that the two phases descend a shared objective the way the Boltzmann machine's $G$ measure did. As the authors state without softening it: "there is no single cost function that is reduced by these two procedures," in part because sleep-phase training fits the recognition weights to data generated by the model rather than to the real training distribution, and in part because the sleep-phase gradient is only an approximation, not the true derivative of $\mathcal{F}$. The one thing that can be said is asymptotic rather than monotone: the recognition and true posterior distributions coincide, in the paper's own notation, only "at the optimal end point, if it can be reached" — with no guarantee en route.
Three claims from the last two sections are checkable rather than merely quotable. First, that the Hebbian outer-product rule Hopfield and Tank restate — $T_{ij}=\sum_s V_i^sV_j^s$ — carves stable wells for a handful of patterns but was never tested by them for how many patterns it can hold before the wells start to merge; that question was flagged in Section 1 as absent from the primary source, not derived, which makes it a rerun rather than a re-derivation. Second, that Ackley, Hinton, and Sejnowski's own admission — "the current version of the learning algorithm is very slow" — can be given an actual number: their own reported benchmark is a 4-2-4 encoder network solved, over "250 different tests," in a median of "I10" (110) learning cycles and a longest run of "18 10" (1,810), each learning cycle requiring a full anneal-to-equilibrium in both a clamped and a free-running phase. Third, that the Helmholtz machine's wake-sleep algorithm was built expressly to remove that annealing at the price of a training procedure with, in the paper's own words, no guaranteed monotone descent. All three are rerun below on 2020s hardware, in every case substituting a task the original authors did not use for the one they did: the Hebbian capacity test uses random binary patterns of length 100 rather than any specific stored image, because Hopfield and Tank stored none for this purpose in the paper itself; and the Helmholtz machine, whose own reported experiments were on shift patterns over 8-pixel rows rather than an encoder, is instead trained on Ackley, Hinton, and Sejnowski's own 4-2-4 encoder task, so that the comparison against the Boltzmann machine is apples to apples on the one problem both algorithms were built to solve.
The Hebbian capacity test stores $p$ random $\pm1$ patterns of length $N=100$ in a Hopfield network by the outer-product rule, zeroes the diagonal, cues the network with each stored pattern corrupted by flipping 20% of its bits at random, and runs asynchronous single-unit updates (each unit set to $\mathrm{sign}(\sum_jT_{ij}s_j)$, in random order, repeated until no unit changes) until the state stops moving. Averaged over five random pattern sets and every stored pattern's own corrupted cue, exact recovery is essentially perfect at $p=5$ ($p/N=0.05$, recall accuracy 1.000) and still high at $p=10$ ($p/N=0.10$, accuracy 0.880), but collapses through the next few points: 0.360 at $p=15$, 0.150 at $p=20$, 0.024 at $p=25$, and exactly zero from $p=30$ on ($p/N \ge 0.30$). Nothing in Hopfield and Tank's paper predicts where this collapse should occur — the 1985 paper restates only the storage rule and the convergence guarantee, not a capacity bound, and Section 1 flagged the merging of wells as a result belonging to "later statistical-mechanics analyses of the model" that are not among this book's primary sources. What the rerun adds is not a citation but a demonstration that the collapse is real, sharp, and falls in the same tenth-of-a-bit-per-unit region ($p/N$ between roughly 0.10 and 0.15) that the wider literature on this exact construction reports; it is a property of the quadratic energy surface and the outer-product rule laid down in Eq. (4), reproducible on a laptop in under a second, and it would not move if the laptop were replaced by a datacenter. Adding compute lets a 2026 machine store and test far more patterns, but it does not change the fraction $p/N$ at which the wells that the Hebbian rule builds start running into each other; that ratio is fixed by the same algebra that fixed the wells in the first place.
The rerun builds a ten-unit Boltzmann machine matching the paper's own 4-2-4 encoder — four visible "input" units and four visible "output" units, both clamped to the same one-hot code during the positive phase (exactly as the paper's Section 4.1 describes, "each environmental vector in turn was clamped over the visible units"), plus two free hidden units — and follows its learning cycle as given: anneal from temperature 20 down through 15, 12, and 10 while gathering the clamped co-occurrence statistic $p_{ij}$, repeat the same schedule fully unclamped for the free-running $p'_{ij}$, then increment or decrement every weight by a fixed step, with sign set by $\mathrm{sign}(p_{ij}-p'_{ij})$. Tested every ten learning cycles against the criterion that all four patterns reconstruct correctly through the two-unit bottleneck, the rerun's network first passed that test at cycle 130 (accuracy 1.0 on that check), after 90 cycles it was already right 92% of the time, and the whole run to convergence took 0.17 seconds of wall-clock time on ordinary 2020s hardware. That cycle count — 130 — lands almost exactly on the paper's own reported figure: a "median... required to discover four different codes was I10 [110] learning cycles," with "the longest time" reaching "18 10 [1,810]" in their runs. The number of learning cycles needed is, in other words, a property of the encoder problem and the annealing-based learning rule, not of the machine executing it, and it reproduced on a laptop to within the same order of magnitude the original authors measured on 1985 hardware. What did not reproduce is the wall-clock cost of those cycles: each one requires, in this rerun's schedule, on the order of a thousand individual stochastic single-unit updates (multiple annealing steps across five clamped-and-unclamped conditions, each itself ten single-unit probes per simulated "time unit"), and even so the entire 130-cycle run finishes in well under a second — the actual multi-day cost the paper's era would have paid for the same number of cycles is simply gone, because each of those thousand-odd individual updates now costs nanoseconds instead of a hand-timed simulation step.
The question Section 2 raised about the Boltzmann machine's slowness was whether annealing to equilibrium is a 1985 hardware ceiling or a property of the Markov chain itself — and a feedforward comparison on the identical problem makes the answer concrete rather than abstract. A conventional 4-2-4 autoencoder — the same four one-hot patterns, the same two-unit bottleneck, but a deterministic sigmoid network trained by ordinary backpropagation rather than Gibbs sampling — reached perfect reconstruction (every pattern's arg-max output correct, with the target unit's activation above 0.9) after 189 epochs of full-batch gradient descent, in 0.0023 seconds. Both networks solve the same problem to comparable precision — the backprop network's final output matrix put 0.90–0.95 on the correct unit and under 0.08 everywhere else, similar in kind to the Boltzmann machine's 87.5% test accuracy over repeated stochastic anneals — but the backprop network needed roughly 74 times less wall-clock time (0.17s against 0.0023s) despite needing more, not fewer, weight-update cycles (189 against the Boltzmann machine's 130) to get there. The gap is not a matter of the two runs happening to use different-sized computers: it comes from what a single "cycle" costs in each scheme. Each backprop epoch is one deterministic forward pass and one deterministic backward pass — two matrix multiplications. Each Boltzmann learning cycle, matched to the paper's own procedure, requires annealing a stochastic chain through four temperatures and then sampling for ten further time-units, separately for the clamped and the free-running phase, over every training pattern — on the order of a thousand individual stochastic unit-updates before a single weight changes. Faster hardware shrinks the constant cost of each of those thousand updates, exactly as it shrinks the cost of a matrix multiplication, but it does not shrink their number, because that number is set by how long the Gibbs chain takes to mix at each temperature on the way down — a property of the energy landscape, not of the processor evaluating each step. This is the same conclusion Section 2 reached from the algebra; the rerun turns it into a wall-clock ratio, measured on one machine, on one problem, with one line separating what changed (whatever constant-factor speedup a modern machine buys per elementary step — this chapter never benchmarks 1985 hardware against the rig used here, so no figure for that generational gap is claimed) from what did not (the fact that a Monte Carlo learning rule needs orders of magnitude more elementary steps than a gradient rule to solve the identical four-pattern problem).
Dayan, Hinton, Neal, and Zemel's own reported experiments trained the wake-sleep algorithm on shift patterns over 8-pixel rows, not on an encoder, so a direct rerun of their numbers is not possible; instead this rerun casts Ackley, Hinton, and Sejnowski's own 4-2-4 encoder as a two-layer generative model — a top layer of two hidden units with learned biases as a prior, a generative mapping down to eight visible units (four input, four output, again clamped together during training), and a recognition mapping the other way — and trains it by wake-sleep exactly as described: a wake phase that samples hidden activity from the recognition weights given a clamped visible pattern and then adjusts the generative weights by the delta rule to make that visible pattern more probable given the sampled hidden cause; a sleep phase that runs the generative model backward from its own prior to produce a fantasy, top to bottom, and adjusts the recognition weights by the same delta rule to predict the hidden sample that produced it. No annealing appears in either phase — one sample per unit per phase, exactly the design Section 2 described. Reaching the same all-four-patterns-correct criterion used for the Boltzmann machine took far more learning cycles — 1,420, against the Boltzmann machine's 130 — but at a small enough per-cycle cost that the whole run still finished in 0.055 seconds, faster in absolute wall-clock terms than the annealed network's 0.17 seconds despite the roughly elevenfold increase in cycle count. The trajectory getting there was not monotone: accuracy checked every twenty cycles rose to 0.5 by cycle 620, fell back to 0.25 at cycle 720, and only climbed reliably past cycle 1,000, before reaching 1.0 at cycle 1,420 and settling to a final rate of 0.900 over thirty repeated stochastic evaluations. That non-monotonicity is not a bug in this implementation; it is the behavior the paper predicts for itself. Dayan and his co-authors state plainly that "there is no single cost function that is reduced by these two procedures," because the sleep phase fits the recognition weights to the generative model's own fantasies rather than to the real data distribution — and a learning curve with no guaranteed monotone objective is exactly what the rerun's dips show.
Laid side by side, the three reruns separate two kinds of claim this chapter has been making, and they separate cleanly. The Hopfield capacity collapse — perfect recall at $p/N=0.05$, failure by $p/N=0.30$ — is a fact about the outer-product rule and the quadratic energy surface it carves; running it on more patterns or faster hardware moves where the curve is measured, not where it bends, because the bend is set by the ratio of stored patterns to units, not by wall-clock time. The non-monotone wake-sleep learning curve is the same kind of fact: no rerun on any hardware makes "there is no single cost function that is reduced by these two procedures" stop being true, because that absence is a property of the algorithm's two-phase construction, not of how fast either phase runs. The wall-clock numbers are the other kind of claim, and they moved by orders of magnitude without changing any ranking: the Boltzmann machine's 130-cycle, 0.17-second convergence on 2020s hardware reproduces almost exactly the cycle count — a "median... 110 learning cycles" — that took the original authors a slow 1985 simulation to observe, and the backpropagation network solved the identical problem in roughly 74 times fewer wall-clock seconds while needing more, not fewer, update cycles to do it. What survives the substitution of a laptop for 1985 hardware is the ordering: gradient descent is cheap per step and needs few steps; wake-sleep is cheap per step but needs many; annealed Boltzmann learning needs relatively few steps but each one is itself a small Markov chain Monte Carlo computation, and that per-step cost — mixing time on an energy landscape with separated wells — is exactly the quantity Section 2 argued would not fall to a faster processor. The rerun does not merely illustrate that argument; it prices it: roughly a thousand elementary stochastic updates per Boltzmann learning cycle against two matrix multiplications per backpropagation epoch, a ratio a 2026 datacenter inherits unchanged from the 1985-vintage hardware the original simulations ran on, because it was never a fact about either machine.
Chapter 2 built a network that computes one static thing: given a fixed input vector, a feedforward stack of weighted sums and squashing functions produces a fixed output vector, and backpropagation assigns credit for the error along paths of finite, known depth. Many of the tasks this book has followed since — a phoneme spread over a hundred milliseconds, a checkers game unfolding over dozens of moves, a sentence generated one word at a time — are not static in this sense; the "input" and "output" are trajectories, and what the network does at time $t$ can legitimately depend on what it or its environment did at every earlier time. The natural way to give a network that capacity is to let some of its own unit outputs feed back in as inputs at the next time step, closing a loop that a purely feedforward architecture does not have. Williams and Zipser open their analysis of one learning procedure for such networks with the claim in its most general form: "Recurrent neural networks can implement dynamical systems of arbitrary complexity. To make use of this ability in cases where the dynamical system is defined only in terms of its input and output we need learning procedures capable of programming recurrent networks." Their own setup fixes the notation this section adopts. A network of $n$ units receives $m$ external input lines; $y(t)$ denotes the $n$-tuple of unit outputs at time $t$ and $x(t)$ the $m$-tuple of external inputs, concatenated into a single $(m+n)$-tuple $z(t)$, with an index set $U$ marking the components of $z$ that are unit outputs and an index set $I$ marking the components that are external inputs. A single weight matrix $W$, of dimension $n\times(m+n)$, then holds every connection in the network at once, with the element $w_{ij}$ the weight on the connection into unit $i$ from either unit $j$ (if $j\in U$) or input line $j$ (if $j\in I$); a bias is folded in as the weight from a constant-valued input line. This is the same weight-and-squashing-function vocabulary as Chapter 2's multilayer perceptron; what is new is only that $z(t)$ can contain a unit's own past output, so that the network's behavior at time $t$ depends on its behavior at time $t-1$, which depended on $t-2$, all the way back to whatever state it was started in.
Williams and Zipser write the network's dynamics for "semilinear units" — the same logistic-style units as Chapter 2 — as a pair of equations indexed over $k\in U$: a net input formed from the concatenated vector, and an output obtained by squashing it. In their notation, unit $k$'s net input at time $t$ is $\sum_j w_{kj}z_j(t)$, and its output at the next step is $y_k(t+1) = f_k(\text{net}_k(t))$, where $f_k$ is that unit's squashing function — "the more compact expression illustrates why we introduced $z$ and the corresponding indexing convention." Crucially, "the external input at time $t$ does not influence the output of any unit until time $t+1$": the network is a discrete-time dynamical system, one step of which is exactly one application of Chapter 2's feedforward equations. The credit-assignment problem this creates is that an error at time $t$ was shaped by the weights not just through the current step but through every earlier one, since $y(t)$ depends on $y(t-1)$, which depends on $y(t-2)$, and so on back to the start of the trajectory. Williams and Zipser attribute the founding idea for handling this to Rumelhart, Hinton, and Williams: "A general framework for the problem was laid out by Rumelhart et al. (1986), who unfolded the recurrent network into a multilayer feedforward network that grows by one layer on each time step. Algorithms based on this concept... might be called backpropagation through time, work well in cases where enough is known about the time structure of the problem to limit the number of layers to some reasonable, fixed value." That closing clause is the limitation the rest of the paper is written against: BPTT presupposes a fixed, previously known horizon over which to unfold, and problems whose temporal extent is not known in advance — Williams and Zipser's own example is strings of arbitrary length generated by a probabilistic finite-state grammar — force the unfolding to be set up with "some a priori knowledge of their maximum length," which is exactly what the real-time algorithm developed later in the paper is built to avoid. The unfolding itself is a bookkeeping device, not a new mathematical object: draw a fresh copy of every unit for each time step from $t_0$ to $t_1$, connect copy $t$ to copy $t+1$ with the network's ordinary recurrent weights, and the resulting graph is a feedforward network of depth $t_1-t_0$ — with the constraint, absent from Chapter 2's networks, that the same weight $w_{ij}$ appears in every copy and so must receive the sum of the error gradient contributed by each. Formally, defining the total error over a trajectory from $t_0$ to $t_1$ as $J_{total}(t_0,t_1) = \sum_{t=t_0+1}^{t_1} J(t)$ for a per-step error measure $J(t)$, the two authors note that "as the trajectory unfolds over time, we can simply accumulate the values of the vector $\nabla J$ at each time step until the final time step" (their "VJ," an OCR artifact of $\nabla J$, restored here) — the gradient of the whole trajectory's error is the sum of per-step gradients, each computed by the same chain rule Chapter 2 already derived, run backward through the unfolded copies instead of through a fixed stack of layers.
Backpropagation through time is not the only way to differentiate a trajectory with respect to a weight, and the alternative is where this chapter's derivations will end up living. Williams and Zipser's real-time recurrent learning algorithm (RTRL) computes the same quantity — the total gradient $\nabla J_{total}(t_0,t_1)$ of Block 2 — without ever building the unfolded copy of the network. Instead of accumulating error backward through a graph whose depth is fixed in advance, RTRL carries forward, at every time step, a sensitivity $\partial y_k(t)/\partial w_{ij}$ for every unit $k$ and every weight $w_{ij}$, updated online from the previous step's sensitivities and the current error $e_k(t)$ as the trajectory runs, so that the weight change can be accumulated cycle by cycle with no need to know in advance where the trajectory will end. These networks "run continually in the sense that they sample their inputs on every update cycle," with no fixed $t_1$ to unfold toward — exactly the property backpropagation through time lacks, and exactly what Williams and Zipser's introduction had identified as missing when "strings of arbitrary length generated by a probabilistic finite state grammar do not have a fixed, previously known length" and so force BPTT to be set up with foreknowledge of a maximum duration. That advantage is bought at a specific, quantified price, and the paper is unusually candid about it: "While the RTRL algorithm is very powerful, it has two significant drawbacks: It requires a great deal of computation on each update cycle, and it is nonlocal. The amount of storage and computation required are independent of time but increase with the number of units in the network. In sequential computers, the algorithm requires, for n units, a storage capacity of roughly n3, and a computation time on each cycle of order n4." Cubic storage and quartic per-step compute in the unit count is a statement about arithmetic operations, not about any particular VAX or Symbolics workstation of 1989 — it is the reason Williams and Zipser confine their own experiments to "networks of moderate size, that is, 20 to 30 units," and the asymptotic cost does not shrink with a faster machine, only the network size it can afford to be run on does. Williams and Zipser's own paper says nothing about how the field would later weigh RTRL against truncated BPTT; but a decade on, Hochreiter and Schmidhuber, introducing LSTM, made exactly that comparison explicit, reporting that their algorithm's "update complexity per weight and time step is essentially that of BPTT, namely, O(1)," and calling this "excellent in comparison to other approaches such as RTRL." That later verdict, not the 1989 paper itself, is the evidence that RTRL's cost — rather than any transient hardware limit — is why truncated backpropagation through time (and its descendants) became the field's working default while RTRL was set aside as a costly full-gradient alternative. The two algorithms this section has now introduced — unfold-and-backpropagate, forward-accumulate-and-update — compute the identical gradient by different bookkeeping, one cheap in space and blind to the future, the other free of any horizon but expensive in units cubed and units to the fourth; the next section derives both in full and follows the credit-assignment problem they share into the vanishing-gradient difficulty that motivates the long short-term memory architecture.
Section 1's unfolding turns a recurrent net into one feedforward copy per time step, and Chapter 2's chain rule runs backward through the unfolded graph exactly as it runs backward through any fixed stack of layers, with the per-step gradients summed because the same weight recurs in every copy. Hochreiter and Schmidhuber's 1997 paper on long short-term memory gives that bookkeeping its standard notation — output unit $k$'s error signal $\vartheta_k(t) = f'_k(net_k(t))(d_k(t)-y_k(t))$, a nonoutput unit $j$'s backpropagated error $\vartheta_j(t) = f'_j(net_j(t))\sum_i w_{ij}\vartheta_i(t+1)$, where $y_i(t)=f_i(net_i(t))$ is a unit's activation and $net_i(t)=\sum_j w_{ij}y_j(t-1)$ its net input — and then reproduces Hochreiter's 1991 analysis of what that recursion does over many steps. Propagating an error from unit $u$ at time $t$ back $q$ steps to unit $v$ scales it by a sum of $n^{q-1}$ terms, each a product over $q$ factors of a squashing-function derivative times a weight: $\partial\vartheta_v(t-q)/\partial\vartheta_u(t) = \sum_{l_1}\cdots\sum_{l_{q-1}}\prod_{m=1}^q f'_{l_m}(net_{l_m}(t-m))\,w_{l_m l_{m-1}}$. Whether that product grows or dies with $q$ depends only on whether its factors sit above or below one in absolute value, and the paper draws the conclusion the field would spend the next decade working around: when every factor exceeds one, the largest term "increases exponentially with q. That is, the error blows up," while when every factor is below one in magnitude, "the largest product decreases exponentially with q. That is, the error vanishes, and nothing can be learned in acceptable time." For the logistic sigmoid, whose derivative never exceeds 0.25, the paper shows the vanishing regime is the one ordinary training lives in — weights below magnitude four keep every factor under one — and shows the obvious escape does not exist: "the use of larger initial weights will not help," since as a weight's magnitude grows without bound the sigmoid derivative at the operating point it induces goes to zero faster than the weight grows. Raising the weights to fight a vanishing gradient drives the unit into the saturated tail of its own squashing function, and the product shrinks anyway.
Hochreiter and Schmidhuber note that "a very similar, more recent analysis was presented by Bengio et al., 1994," and it is worth pausing on that second derivation because it reaches the same wall by an entirely different road, which is exactly what makes the wall structural rather than incidental. Bengio, Simard, and Frasconi frame the question not in terms of a specific weight matrix but in terms of what any parametric dynamical system must do to hold a bit of information robustly against noise: they define a system as "robustly latched" at an attractor when small, irrelevant perturbations to the input cannot dislodge the state from that attractor's basin, and prove, as their Theorem 4, that "If the input ut is such that a system remains robustly latched on attractor X after time 0" the derivative of the current state with respect to the state many steps in the past goes to zero as the gap grows. Their own gloss removes any ambiguity about what this means for training: "when storing one or more bit of information in a way that is resistant to noise, the gradient with respect to past events rapidly becomes very small in comparison to the gradient with respect to recent events." The paper's own abstract states the trade-off this forces in its most general form, independent of architecture, unit type, or learning rate: the results "expose a trade-off between efficient learning by gradient descent and latching on information for long periods," and the introduction sharpens it into a dichotomy — a system is "either such a system is stable and resistant to noise or, altematively, it is efficiently trainable by gradient descent, but not both." This is the central question this book keeps returning to, answered here in the negative in the strongest available sense. Hochreiter's product-of-derivatives argument and Bengio, Simard, and Frasconi's attractor argument are independent proofs of the same fact from different premises, and neither depends on the width of a bus, the clock speed of a workstation, or the number of floating-point units on a chip; both are statements about what a gradient computed by the chain rule must do when a system is asked to remember reliably across many steps. A faster computer lets you run the same exponentially decaying product on a bigger network or a longer sequence, and the product still decays. Overcoming this required a change to the architecture computing the gradient, not a change to the hardware computing it — which is exactly what the long short-term memory cell, derived next, supplies.
Hochreiter and Schmidhuber's fix starts from the smallest case that can possibly avoid the decay just proven: a single unit $j$ with one connection to itself. Its local error backflow is $\vartheta_j(t) = f'_j(net_j(t))\vartheta_j(t+1)w_{jj}$, and "to enforce constant error flow through j, we require $f'_j(net_j(t))w_{jj} = 1.0$" — integrating that condition forces $f_j$ to be linear and $w_{jj}$ to equal one, a self-connected linear unit with weight exactly 1, which the paper calls the constant error carousel (CEC): a unit whose error, once inside, is neither amplified nor attenuated by a single further time step, because there is no derivative less than or greater than one left in the loop to multiply it by. But a CEC that is only ever connected to itself is useless; it must also receive input and send output, and doing so through ordinary weights reopens two conflicts intrinsic to any gradient-based scheme, not just this one. The first is what the paper names the input weight conflict: with a single incoming weight $w_{ji}$, "assume that the total error can be reduced by switching on unit j in response to a certain input and keeping it active for a long time (until it helps to compute a desired output)" — the same weight must both open the gate for the relevant input and stay closed against every later, irrelevant one, and gradient descent pulls it in both directions at once, so that "this conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling write operations through input weights." The second is the mirror image on the way out, the output weight conflict: with $j$ already storing something useful, a single outgoing weight $w_{kj}$ must release that content when it is wanted downstream and suppress it otherwise, and the paper again concludes this "conflict makes learning difficult and calls for a more context-sensitive mechanism for controlling read operations through output weights." Both conflicts get sharper, not milder, as the lag between storage and use grows — exactly the regime where the constant error carousel is supposed to help — because a fixed weight has no way to condition its behavior on how much time has passed. The remedy is to stop asking one static weight to do a time-varying job and instead let the network compute, at every step, a separate multiplicative signal that decides whether the CEC should listen or speak: "to avoid input weight conflicts, $in_j$ controls the error flow to memory cell $c_j$'s input connections $w_{c_ji}$. To circumvent $c_j$'s output weight conflicts, $out_j$ controls the error flow from unit j's output." The resulting unit, the memory cell, wraps the linear CEC in exactly two such gates. Writing $y_{in_j}(t)$ and $y_{out_j}(t)$ for the input and output gate activations at time $t$ and $net_{c_j}(t)$ for the cell's ordinary net input, the cell's internal state and output are $s_{c_j}(0) = 0$ and, for $t>0$, $s_{c_j}(t) = s_{c_j}(t-1) + y_{in_j}(t)\,g(net_{c_j}(t))$, with output $y_{c_j}(t) = y_{out_j}(t)\,h(s_{c_j}(t))$, where $g$ squashes the candidate content being written and $h$ squashes the content being read out. The self-recurrence on $s_{c_j}(t-1)$ carries coefficient exactly one — the CEC is still in there, untouched — so information written into the cell at one step neither grows nor shrinks on its own as it waits; what changes over time is only how much of it the two sigmoidal gates let in or out, and those gates, unlike the fixed weight they replace, are themselves functions of the current input and can learn to open only when opening is useful and close otherwise, which is precisely the context-sensitivity the two conflicts demanded.
The training rule the 1997 paper attaches to this architecture makes the same point in a different form. Rather than backpropagate the full BPTT recursion through the gates and the CEC alike, Hochreiter and Schmidhuber cut the recursion deliberately: "With the efficient, truncated update rule, error flows only through connections to output unit, and through fixed self-connections within cell blocks... Error flow is truncated once it 'wants' to leave memory cells or gate units." Gradients into the gates and into the cell's input use only the current time step's contribution, not the chain back through all previous openings and closings of the gate; the only signal allowed to survive across arbitrarily many steps unmolested is the one running through the self-connection of weight one. The paper is explicit that this approximation costs nothing in the direction that matters — the truncation drops terms that would themselves be vanishingly small in the conventional analysis, while leaving intact the one channel, the CEC's fixed self-loop, whose derivative is exactly one at every step by construction and therefore never had anything to vanish. The gates make training tractable in the same stroke that they make the memory usable: because the gate values in general do not train against an exponentially decaying signal (they only need to be right for the current step, not have their long-past history explained to them by backpropagation), the network can learn when to write and when to read using an ordinary, local gradient, while what it stores rides the CEC undamaged for as many steps as the gates choose to hold it closed. Three years later, Gers, Schmidhuber, and Cummins found the one place this construction breaks on its own terms, in a setting Hochreiter and Schmidhuber had not tested: continuous, unsegmented input streams rather than sequences with an externally supplied reset. Because the CEC's self-weight is exactly one and the input gate only adds, never subtracts, "a continual input stream eventually may cause the internal values of the cells to grow without bound, even if the repetitive nature of the problem suggests they should be reset occasionally." Their fix, the forget gate, extends the same gating logic to the self-connection itself: a forget-gate unit computes its own activation from the current input exactly as the input and output gates do, and "the revised update equation for sc in the extended LSTM algorithm is (for t > 0): scv j (t) = yϕj(t) scv j (t −1) + yinj(t) g(netcv j (t))" — the fixed coefficient of one on the recurrence is replaced by a learned, input-dependent coefficient in [0,1] that the network can drive toward one to preserve the constant error carousel indefinitely, or toward zero to reset it, on its own schedule, exactly as "Forget gates learn to reset memory cell contents once they are not needed anymore." Nothing about this repair changes the argument of the previous section; it is a further instance of it. The vanishing-gradient proofs of Hochreiter (1991) and Bengio, Simard, and Frasconi (1994) constrain what a fixed, externally imposed recursion can do to an error signal over many steps; the LSTM cell answers by making the recursion's own coefficient a learned quantity instead of a fixed weight or a sigmoid derivative, so that the network, rather than the architecture, decides when the product in that recursion should sit at one. That is an architectural change to what is being differentiated, not a numerical trick for computing the same gradient faster or on bigger hardware, and it is why a machine with the memory and clock speed of 2026 does not make BPTT converge on the tasks the 1997 paper reports RTRL and BPTT failing at outright — the failure was never about how long the multiplication took, but about what the multiplication computed.
Three claims from the last two sections can be run rather than merely quoted. First, that a product of $q$ derivative-and-weight factors shrinks geometrically as $q$ grows whenever every factor is below one in magnitude — the arithmetic Section 2 attributed to Hochreiter's 1991 analysis and to Bengio, Simard, and Frasconi's independent attractor argument. Second, that Bengio, Simard, and Frasconi's own "minimal task" — a single recurrent neuron asked to latch the sign of an early input against later Gaussian noise, exactly the construction their Section III calls "latching one bit of information, represented by the sign of its activation" — runs into exactly the wall their own experiments report: "When T becomes large it is extremely difficult to attain convergence." Third, that the long short-term memory cell removes this difficulty by construction, solving latching-style problems at lags Hochreiter and Schmidhuber report in the hundreds of steps, on tasks where they state that "BPTT and RTRL already fail in case of 10-step minimal time lags." All three are rerun below on 2020s hardware, using a single scalar recurrent unit and a single memory cell — matching the "single recurrent neuron" scale Bengio, Simard, and Frasconi themselves used for this exact task, not the larger networks either paper's harder experiments employed — trained on a synthetic latching task built to their own specification: a class-defining symbol at the sequence's start, pure Gaussian noise for every step after it, and a decision read out only at the end. A fourth question, absent from either the 1994 or the 1997 paper because the tool did not yet exist, is added deliberately: whether an adaptive optimizer unavailable to either set of authors (Adam, 2014) moves the failure boundary, which is this book's standing test of whether a reported limitation is a fact about the mathematics or an artifact of the hardware and software available at the time.
The first rerun is the smallest possible test of the product-of-derivatives argument itself: propagate a unit error backward through a single tanh unit's own self-recurrence, step by step, and watch what happens to its magnitude as the number of steps grows. With the recurrent weight fixed at $a=0.9$ and the unit driven by random inputs so its operating point is realistic rather than degenerate, the injected gradient of magnitude 1.0 at the final step falls to 0.251 five steps back, 0.035 at ten steps, 0.0014 at twenty, and 0.00000036 at thirty — a geometric collapse, matching the claim that "the largest product decreases exponentially with q" and that beyond a certain lag "nothing can be learned in acceptable time." Raising the recurrent weight to $a=1.5$, the value the paper's argument singles out as the tempting fix, makes the collapse worse rather than better: 0.0065 at five steps, 0.000047 at ten, and at the floating-point floor ($10^{-8}$) by twenty — confirming in a live computation the paper's own diagnosis that "the use of larger initial weights will not help," because a larger weight drives the tanh unit further into saturation, and the derivative lost to saturation more than offsets whatever the larger weight would otherwise contribute. Against both cases, the long short-term memory cell's constant error carousel — a unit with squashing function the identity and self-weight exactly 1 — was probed the same way: with its single gate held open, the same injected gradient measures 1.0 at every lag tested, from five to forty steps back, undiminished to eight decimal places, because there is no factor in that loop's product that is not exactly one. Nothing about this comparison depends on which processor ran the loop; a product of numbers below one shrinks whether the multiplications happen on a 1997 workstation or a 2026 accelerator, and a self-loop of coefficient one is unaffected by either.
The rerun's task follows Bengio, Simard, and Frasconi's own construction step for step. Their three conditions for a system that can learn long-term dependencies require "that the system be able to store information for an arbitrary duration," "that the system be resistant to noise," and that its parameters be trainable in reasonable time; their minimal task realizes this by making a sequence's class depend only on its first symbols while every later input, in their notation, "$u_t$ is zero-mean Gaussian noise for $t>L$" — noise the system must ignore without letting it erase what was stored. This rerun's version sets $L=1$: a single scalar $x_0=\pm1$ carries the class, followed by $T-1$ steps of Gaussian noise ($\sigma=0.2$), read out through a sigmoid at step $T$ and trained by ordinary binary cross-entropy — a simplification of their $\pm0.8$-target, tunable-input-window design to the smallest case that still isolates the same difficulty. A single-tanh-unit recurrent network, trained by full backpropagation through time (no truncation — feasible here because $T\le100$) with plain gradient descent, learns the task perfectly at $T=2$ (test accuracy 1.000, cross-entropy loss 0.018) but collapses to chance at every longer lag tested: accuracy 0.521 at $T=10$, 0.491 at $T=30$, and 0.494 at $T=60$, with the loss stuck at 0.693 — the entropy of a coin flip — in each case. This is the numerical form of "when $T$ becomes large it is extremely difficult to attain convergence," reproduced with a different (BCE rather than squared-error) loss and different training data, on a laptop rather than the workstation the 1994 paper's simulations ran on, and it appears at the same short lag ($T=10$) where Hochreiter and Schmidhuber's own comparison sets in.
The same task, run on a single long short-term memory cell — one input gate, one output gate, one cell input, trained by the full gated backpropagation Section 2 derived, with no truncation — was not an immediate success. With gate weights initialized the same way as the plain unit's (small, zero-mean, zero bias) and trained by plain gradient descent, the cell solved $T=2$ and $T=10$ perfectly but collapsed to chance by $T=30$ (accuracy 0.488) and stayed there through $T=100$. The cause was not the vanishing-gradient product Section 2 proves the constant error carousel immune to — the carousel's self-loop is exactly 1 whether the cell is one step old or a hundred — but a different failure the carousel's own construction invites: with the input gate open by default and nothing to close it, the cell state $s_c(t)=s_c(t-1)+y_{in}(t)g(net_c(t))$ accumulates a little noise at every one of the $T-1$ irrelevant steps, and by $T=30$ or $T=100$ that accumulated drift has driven $\tanh(s_c)$ into saturation before the class-bearing signal is ever read out — the same growth-without-bound failure mode Gers, Schmidhuber, and Cummins would name three years later for continuous streams, where they warn that "a continual input stream eventually may cause the internal values of the cells to grow without bound." Two changes, neither available to either 1994 or 1997 authors as a turnkey tool, fixed it: initializing the input gate's bias strongly negative (so the gate starts closed and must learn to open only at the one step that matters) and replacing plain gradient descent with the Adam optimizer. With both changes, the cell reached accuracy 1.000 at $T=10$ and $T=30$, and 1.000 at $T=60$; at $T=100$ it needed a more negative bias still ($-3.0$) and 4,000 epochs to reach accuracy 1.000 (loss 0.0040), matching, at a tenth of the scale, Hochreiter and Schmidhuber's own hardest reported case — "task 2c," with "minimal time lags of 1000 steps," which "LSTM solves" while "BPTT and RTRL already fail in case of 10-step minimal time lags" — confirmed here at $T=100$ with a single cell rather than their larger networks.
The natural objection to Block 3's failure is that it used the vanilla gradient descent of 1994, not the adaptive per-parameter optimizers of three decades later — so the comparison was rerun a second time, giving the plain tanh unit the identical Adam optimizer and epoch budget (3,000 epochs) given to the memory cell. The result moves the boundary without erasing it. Adam lets the single recurrent unit solve $T=10$ (accuracy 1.000, loss 0.0001) and even $T=30$ (accuracy 1.000, loss 0.0001) — lags where plain gradient descent had already collapsed to chance — but at $T=60$ and $T=100$ the same optimizer, given the same budget, lands back at chance (accuracy 0.523, loss 0.693 at both lags). A better optimizer, unavailable to Bengio, Simard, and Frasconi in 1994, buys the plain architecture roughly a six-fold increase in the lag it can bridge — from about 10 steps to about 30 — which is a genuine, quantifiable answer to this book's standing question about hardware and software progress. But it is a partial answer, not a refutation: the long short-term memory cell, given nothing but the same Adam optimizer and no architectural advantage beyond the constant error carousel and its two gates, reaches $T=100$ where the plain unit — with identical optimization machinery — cannot get past $T=30$. The product-of-derivatives argument in Block 2 does not care which optimizer chases its exponentially small signal; Adam's per-parameter rescaling can compensate for a gradient that is small but non-zero, buying some number of additional steps, but it cannot compensate for a gradient that has decayed to the floating-point neighborhood of zero, which is what a geometric decay eventually reaches for any fixed rate short of exactly one. The rerun does not show that faster optimization is powerless against vanishing gradients; it shows precisely how much it buys (a small constant factor in the tolerable lag) and precisely where that factor runs out, leaving the architectural fix to do the rest.
Laid side by side, the reruns separate the same two kinds of claim earlier chapters' reruns have separated. The geometric decay of a backpropagated error through a self-recurrent unit — measured directly in Block 2, at $10^{-8}$ by twenty steps for a weight of 1.5 — is a fact about multiplying numbers below one together; it does not move if the multiplication is carried out by a 1994 workstation, a 2020s laptop, or a 2026 datacenter, because the number of factors and their magnitudes are fixed by the architecture and the operating point, not by the machine. The constant error carousel's immunity to that decay — a gradient of magnitude 1.0 at every lag from five to forty steps, to eight decimal places — is the same kind of fact in reverse: a self-loop of coefficient exactly one has nothing in its product for any processor to shrink. Where hardware and, more precisely, three decades of optimizer research actually did move the needle is the practical boundary of the plain architecture: vanilla gradient descent's failure at $T=10$, reported by Bengio, Simard, and Frasconi as occurring already at short lags and confirmed here in Block 3, becomes a failure at $T=60$ once Adam replaces it in Block 5 — real progress, bought by forty years of optimization research rather than by raw compute, and worth stating plainly rather than folding into either "nothing changed" or "everything changed." What did not move, on any optimizer this rerun tried, is the ordering between the two architectures: at every lag long enough to matter, the memory cell — solving $T=100$ where the tuned plain unit still cannot pass $T=30$ — wins not because it is trained differently but because Section 2's constant error carousel gives it a channel along which the gradient literally cannot decay, a guarantee no choice of optimizer can manufacture for an architecture that lacks it.
Volume I's narrative told this as a story of one criterion giving way to another: Mitchell's version spaces demanded that a learned hypothesis match every training example exactly, and Valiant's 1984 framework replaced that demand with something weaker and, it turned out, formalizable. This section restates the replacement in the notation the original papers used, because everything the rest of this chapter derives — the VC dimension, the sample-complexity bounds, structural risk minimization — is an answer to one formal question, posed with full precision twice, independently, a decade and a half apart: once by Vladimir Vapnik and Alexey Chervonenkis in Moscow in the theory of pattern recognition, and once by Leslie Valiant in Cambridge, Massachusetts, in the theory of computation. Both traditions began from the same primitive situation. A learner sees a finite sample of labeled instances drawn at random from some space of possible inputs. It must produce, from that sample alone, a rule that will be judged not against the sample it saw but against future instances drawn from the same source. Nothing about the sample tells the learner, in itself, whether the rule it extracts will hold up — a rule that perfectly reproduces every training label is compatible with wild disagreement on the next instance, unless something more is assumed or proved. The two traditions differ in the vocabulary they built to make this precise: Valiant's descendants speak of concepts, instance spaces, and hypothesis classes; Vapnik's speak of risk functionals, loss, and probability measures over pairs. Both sections below present a tradition's own notation rather than translating one into the other prematurely, because the convergence of the two vocabularies onto the same combinatorial quantity — the VC dimension — is itself the content of the next section, and is best appreciated by having watched both roads arrive at the same crossing from different directions.
Blumer, Ehrenfeucht, Haussler, and Warmuth's 1989 restatement of Valiant's framework gives the vocabulary this chapter uses throughout. "The essential idea consists of approximating an unknown 'concept' from a finite number of positive and negative 'examples' of the concept" — a concept being nothing more than a subset c of a fixed instance space X ("Let X be a set which is assumed to be fixed and known. X is sometimes called the instance space"), and a labeled example being a pair (x, 1) or (x, 0) recording whether x falls inside c or outside it. A concept class C is a collection of such subsets — the halfspaces of the plane, say, or the Boolean formulas expressible in k terms of n variables — and a hypothesis class H is the space of candidate rules the learner is permitted to output; the two need not coincide. "The examples are assumed to be drawn according to some probability distribution, and the same distribution is used to evaluate how well a concept is learned," and critically "no assumptions are made about which particular distribution is used": the guarantee sought is not for one convenient P but for every P on X whatsoever. Given a sample of m labeled examples generated this way, "the learner outputs a hypothesis which is his current estimate of the concept. The error of the estimate is taken as the probability that the hypothesis will incorrectly classify the next randomly chosen example" — that is, error(h) = P(h Δ c), the probability mass, under the same distribution P that generated the training sample, of the symmetric difference between hypothesis and target. Valiant's criterion, cleaned here of the scanned original's OCR-garbled epsilons and deltas but not of its content, is this: the pair (C, H) is learnable if there is a learning algorithm such that for every accuracy ε > 0 and confidence δ > 0 there exists a sample size m for which, whatever the target concept c ∈ C and whatever the distribution P on X, a random m-sample drawn according to P yields, with probability at least 1 − δ, a hypothesis of error less than ε. The last clause is the one easiest to read past and hardest to give up: "the number of examples required for a given accuracy and confidence should be independent of the distribution from which the examples are drawn." A bound on m that happened to work for uniform distributions but blew up on some pathological P would not be a PAC bound at all — the whole point of the framework is a guarantee that holds before the distribution is known, not after.
Vapnik's own vocabulary, as Burges lays it out in the notation Vapnik's books use, starts from a random pair rather than a deterministic concept. Each observation is a pair (xi, yi), a vector and its label, and "the data are assumed 'iid' (independently drawn and identically distributed)" according to some fixed but unknown P(x, y) — a strictly more general setup than Valiant's, since it allows a given x to carry different labels on different draws, with only the conditional distribution of y given x held fixed. A learning machine is a family of candidate functions f(x, α) indexed by a parameter α — "the functions f(x, α) themselves are labeled by the adjustable parameters α," so that fixing α selects one member of the family, exactly as fixing a hypothesis h selects one member of H above. The quantity a learner actually cares about is the expected loss of the trained machine over the whole distribution: "the quantity R(α) is called the expected risk, or just the risk" — Vapnik's actual risk, the direct analogue of Valiant's error(h). Because P(x, y) is unknown, R(α) cannot be computed; the only accessible surrogate is "the 'empirical risk' Remp(α)," "defined to be just the measured mean error rate on the training set" — the fraction of the m training pairs the machine gets wrong, the direct analogue of consistency with a sample in the PAC setting. Burges's own gloss on why neither of the naive extremes will do is worth keeping, because it recurs, dressed in VC-dimension notation, throughout the rest of this chapter: "a machine with too much capacity is like a botanist with a photographic memory who, when presented with a new tree, concludes that it is not a tree because it has a different number of leaves from anything she has seen before" — a hypothesis class rich enough to fit any sample perfectly is, for exactly that reason, unable to say anything reliable about the next one.
Both vocabularies name the same act: pick a member of a restricted space of candidate rules that agrees with, or scores well on, a finite sample, and hope the agreement extrapolates. Both also inherit the same warning, which motivates the entire enterprise of measuring hypothesis-space size. If a learner places no restriction whatsoever on the space of hypotheses it will consider, and has no preference among hypotheses that fit the sample equally well, then every classification of the unseen instances remains equally possible given the sample, and, as Haussler puts it, "no inductive method can do better on average than random guessing." Restricting the hypothesis space — to linear separators, to conjunctions of at most k terms, to functions realizable by a bounded network — is not an approximation forced on the learner by weak hardware; it is the only thing that makes generalization from a finite sample logically possible at all. "This is known as inductive bias," and the question this chapter answers is how to price it: not "is the hypothesis space small enough," which has no distribution-free answer in terms of raw cardinality once X is infinite, but "how large is the hypothesis space in the one combinatorial sense that governs both Valiant's sample complexity and Vapnik's gap between R(α) and Remp(α) at once." That both traditions arrive at the same number — the Vapnik-Chervonenkis dimension — by asking questions that look nothing alike on the surface (a worst-case count of consistent hypotheses in one case, a uniform-convergence rate for empirical frequencies in the other) is the fact the next section derives from each side's own definitions, rather than assumes.
Begin on Valiant's road, in the notation the source itself uses when it restates Blumer, Ehrenfeucht, Haussler, and Warmuth's definitions "following closely the notation of [6]." Fix an instance space X and a concept class C ⊆ 2^X. For any finite subset S of X, look at the traces C leaves on S — the collection of sets S ∩ c as c ranges over C. The text's own definition, garbled by scanning but not in substance: "Definition 2 (Vapnik-Chervonenkis Dimension) Let C C 2X . For any finite set S C X, let ec (S) = {S n c : c E C}. S is said to be shattered by C if c (S) = 2S . The Vapnik-Chervonenkis dimension of C is defined to be the largest integer d for which there exists a set S C X of cardinality d such that S is shattered by C. If no such largest integer exists then the VC dimension of C is infinite." Read past the OCR — "C C 2X" is C ⊆ 2^X, "ec(S)" is the trace Π_C(S) = {S ∩ c : c ∈ C}, and "c (S) = 2S" is the condition Π_C(S) = 2^S, that every one of the 2^|S| possible labelings of S is realized by some concept in C. The VC dimension d of C is the largest cardinality of a set that C can shatter this completely; if arbitrarily large finite sets can be shattered, d = ∞. Nothing here mentions probability, error, or sample size — the definition is purely combinatorial, a fact about how richly C can carve up finite subsets of X, decided before any distribution or learning algorithm enters the picture.
The result Blumer, Ehrenfeucht, Haussler, and Warmuth proved is that this one number decides learnability outright, in both directions — but the theorem has two clauses, not three, and it is numbered 2.1, not 1. Stated exactly from the paper: "THEOREM 2.1. Let C be a nontrivial, well-behaved 3 concept class. (i) C is untformly learnable tfand only if the VC dimension of C is finite. (ii) Ifthe VC dimension of C is d, where d < ~0, then (a) for 0 < E < 1 and sample size at least any consistent function A: Se -+ C is a learning function for C and (b)4 for 0 < e < $ and sample size less than max e In i, d(1 - 2(t( 1 - 6) + 6)) , no function A: Se + H, for any hypothesis space H, is a learning function .for C." Clause (i) is the qualitative iff — finite VC dimension exactly separates the learnable classes from the unlearnable ones — and clause (ii) is the quantitative refinement built on top of it: given that d is finite, (ii)(a) is the sufficiency half and (ii)(b) the necessity half, together pinning down the sample complexity m(ε, δ) up to constants entirely in terms of d, ε, and δ, with no dependence on the distribution P or on which c ∈ C is the true target. The scanning has done real damage here, and of two different kinds. In (ii)(a) it has dropped the sample-size expression outright — the sentence jumps from "sample size at least" straight to "any consistent function," with the bound itself missing rather than merely garbled. In (ii)(b) the bound survives but scrambled: "max e In i, d(1 - 2(t( 1 - 6) + 6))" is max((1-ε)/ε · ln(1/δ), d(1-2(ε(1-δ)+δ))) with ε and δ rendered as bare letters — confirmed by the paper's own restatement of this same quantity in the proof, "m = max(( 1 - c)/cln l/6, d( 1 - 2(~( 1 - S) + 6)))," where "c" again stands for ε and "S" for δ. The missing closed form for (ii)(a) is not lost, though: earlier in the paper, describing this same result before the formal statement, the authors give it explicitly — "using sample size m(c, 6) = max(4/E log(2/6), 8d/c log( 13/t))" — which decodes to the standard form reproduced in every textbook that cites this theorem: m(ε, δ) = max((4/ε) log(2/δ), (8d/ε) log(13/ε)) examples suffice for any algorithm that returns a hypothesis consistent with the sample, and roughly that many are also necessary by (ii)(b). The structure of the bound is the payoff of Definition 2: sample complexity grows only linearly in d (up to the log factor), so a concept class can have an infinite instance space, an uncountable hypothesis class, and still be learnable from a sample size that does not grow with either — provided its VC dimension is finite. This is the distribution-free half of the convergence promised at the end of the last section: a worst-case combinatorial quantity, defined by counting shatterable sets, converts directly into a worst-case bound on how many examples a consistent learner needs.
Now take Vapnik's road, in the notation Burges uses to present it, and watch it arrive at the same crossing from the opposite direction. Where Valiant's tradition starts from a fixed target concept and asks how many examples suffice to approximate it, Vapnik's tradition starts from the gap between two already-defined quantities — the actual risk R(α) of §6.1's setup and its empirical surrogate Remp(α) — and asks how large that gap can be, with what confidence, as a function of sample size alone. The result is a single inequality: "Now choose some η such that 0 ≤ η ≤1. Then for losses taking these values, with probability 1 −η, the following bound holds (Vapnik, 1995): R(α) ≤Remp(α) + sµh(log(2l/h) + 1) −log(η/4) l ¶ (3)" — the scanner's "sµ...¶" standing for a square root bracket, so that the readable form is R(α) ≤ Remp(α) + sqrt[(h(log(2l/h)+1) − log(η/4))/l], where l is the number of training observations, η is the confidence parameter (the bound holds with probability at least 1−η), and h is "a non-negative integer called the Vapnik Chervonenkis (VC) dimension, and is a measure of the notion of capacity mentioned above." Burges names the second term on the right "the “VC confidence.”" Three things about the bound are worth holding onto in exactly the words used to establish them: it is "independent of P(x, y)" — no assumption on the data-generating distribution beyond the same iid draw used in §6.1 — it cannot usually be evaluated on the left because R(α) is unknown, and it can always be evaluated on the right because h, l, and η are known quantities. Minimizing the right-hand side over a family of candidate machines, rather than minimizing empirical risk alone, is, in Burges's words, "the essential idea of structural risk minimization." The mechanism for doing so requires nesting: "Note that the VC confidence term in Eq. (3) depends on the chosen class of functions, whereas the empirical risk and actual risk depend on the one particular function chosen by the training procedure. We would like to find that subset of the chosen set of functions, such that the risk bound for that subset is minimized" — divide one large hypothesis class into an ordered sequence of nested subsets of increasing VC dimension, train within each, and pick the subset whose sum of empirical risk and VC confidence is smallest, rather than the largest subset that fits the sample. The quantity h in Eq. (3) is not a second, independent notion of capacity invented by the statistical tradition — it is Definition 2 of the last section, restated from scratch and arriving at the identical combinatorial object. Burges's own definition of shattering, given before he ever writes Eq. (3), reads: "if a given set of l points can be labeled in all possible 2l ways, and for each labeling, a member of the set {f(α)} can be found which correctly assigns those labels, we say that that set of points is shattered by that set of functions. The VC dimension for the set of functions {f(α)} is defined as the maximum number of training points that can be shattered by {f(α)}." Strip away the vocabulary — "points" for instances, "the set {f(α)}" for the hypothesis class, "labeled" for the sample's classification — and this is Blumer, Ehrenfeucht, Haussler, and Warmuth's Π_C(S) = 2^S with C renamed {f(α)} and S renamed the l points: the same closed loop of quantifiers (a set of size h all of whose 2^h labelings are achievable by some member of the class), decided before any distribution or sample enters. Burges even reproduces, as a worked example, the same computation the last section left as a bare corollary: three points in the plane can always be shattered by oriented lines but four never can, "the VC dimension of the set of oriented lines in R2 is three," and the general theorem behind it — "the m points can be shattered by oriented hyperplanes if and only if the position vectors of the remaining points are linearly independent" — gives the VC dimension of oriented hyperplanes in Rⁿ as n+1 by the same shattering criterion Definition 2 uses for arbitrary concept classes. What differs between the two roads is not the object being measured but the question asked of it once measured. Blumer, Ehrenfeucht, Haussler and Warmuth fix the accuracy ε and confidence δ in advance and solve for the sample size m(ε, δ) = max((4/ε) log(2/δ), (8d/ε) log(13/ε)) that a consistent learner needs — a bound on m in terms of d. Vapnik's bound, as Burges gives it, runs the same relationship in the other direction: fix the sample size l and the confidence 1−η already spent, and solve for the achievable gap between empirical and actual risk in terms of h — a bound on R(α) − Remp(α) in terms of the same VC dimension, now called h rather than d only because the two papers chose different letters. Both bounds are driven by the identical asymptotic quantity — a VC dimension divided by a sample size, appearing inside a logarithm of their ratio — because both are proved by the identical technical device underneath the two papers' surface vocabularies: a uniform bound on how much empirical frequencies, taken over every hypothesis in a class simultaneously, can deviate from the true probabilities that generated them, with the deviation controlled by how many ways the class can label a finite sample rather than by how many hypotheses it contains outright. Valiant's computational-learning-theory descendants derived this by counting consistent hypotheses directly; Vapnik and Chervonenkis derived it, a decade and a half earlier, from the theory of empirical processes, asking when a supremum over a possibly infinite hypothesis class of the difference between empirical and true frequencies still converges to zero. Both proofs bottom out in exactly the combinatorial fact Definition 2 states: what makes the supremum over an infinite class behave like a supremum over a finite one is that the class's trace on any finite sample of size m is bounded not by the (possibly infinite) cardinality of the class but by a polynomial in m once m exceeds the VC dimension — the fact Blumer, Ehrenfeucht, Haussler and Warmuth prove as their Proposition A2.1, following Vapnik and Chervonenkis's own earlier bound on the growth function, and the reason a hypothesis class with finite VC dimension behaves, for sample-complexity purposes, as if it were finite even when it is not. Two vocabularies, two decades, two continents, and one number.
Two of the claims made in the derivation above are checkable directly, without appeal to Burges's or Blumer, Ehrenfeucht, Haussler and Warmuth's proofs, because they are finite combinatorial and arithmetic facts rather than asymptotic ones. The first is Theorem 1's corollary: "the VC dimension of the set of oriented lines in R2 is three," and more generally the VC dimension of oriented hyperplanes in Rⁿ is n+1, "since we can always choose n + 1 points... such that the position vectors of the remaining n points are linearly independent, but can never choose n + 2 such points." This is not, however, a claim about every possible finite point set: Burges is explicit, right after defining shattering, that "if the VC dimension is h, then there exists at least one set of h points that can be shattered, but it in general it will not be true that every set of h points can be shattered." The corollary trades on that asymmetry. Its shatterable half is existential — "we can always choose n + 1 points... such that the position vectors of the remaining n points are linearly independent" asserts that some configuration of n+1 points works, not that every configuration does; three collinear points in R², for instance, have VC dimension 3 as a class but are themselves an n+1 = 3 point set that cannot be shattered, because a collinear middle point cannot be separated from its two neighbors by an oriented line without splitting one of them off too. Its unshatterable half, "can never choose n + 2 such points," is universal, because n+2 points in Rⁿ can never all have linearly independent position vectors relative to any choice of origin — that is a dimension-counting fact true of every configuration, degenerate or not. What is checkable by brute-force enumeration, then, is two different things: existence of a good (non-degenerate, "general position") n+1-point configuration, and impossibility for every n+2-point configuration. The rerun below draws random point sets in R² and R³ — which, being continuous draws, land in general position with probability 1 and so are never collinear or otherwise affinely degenerate — and for each one attempts, by training a linear support vector classifier with a very large penalty (a stand-in for exact linear separability) on every possible labeling of the points, to determine whether that configuration is shattered. <run_code label="vc-dim-hyperplanes-shattering"> import numpy as np from sklearn.svm import SVC from itertools import product rng = np.random.default_rng(0) def can_shatter(points, n_dims): k = len(points) for labeling in product([-1, 1], repeat=k): y = np.array(labeling) if len(set(y)) == 1: continue clf = SVC(kernel='linear', C=1e6) clf.fit(points, y) pred = clf.predict(points) if not np.all(pred == y): return False, labeling return True, None def trial(n_points, n_dims, n_trials): shattered_count = 0 for _ in range(n_trials): pts = rng.normal(size=(n_points, n_dims)) ok, _ = can_shatter(pts, n_dims) if ok: shattered_count += 1 return shattered_count, n_trials for npts in [3, 4]: ok, total = trial(npts, n_dims=2, n_trials=100) print(f"R^2, {npts} random points: shattered in {ok}/{total} trials") for npts in [4, 5]: ok, total = trial(npts, n_dims=3, n_trials=60) print(f"R^3, {npts} random points: shattered in {ok}/{total} trials") </run_code> The run printed: R² with 3 random points shattered in 100 of 100 trials, R² with 4 random points shattered in 0 of 100 trials; R³ with 4 random points shattered in 60 of 60 trials, R³ with 5 random points shattered in 0 of 60 trials. This confirms both halves of Theorem 1's corollary at n=2 and n=3, but not as a single symmetric boundary. The n+1 result (100/100, 60/60) confirms the existential claim for generic configurations: every non-degenerate random draw of n+1 points happened, as expected, to be a "some configuration exists" witness, because almost-surely no random draw is collinear or otherwise affinely dependent — it says nothing about the (measure-zero, but real) degenerate n+1-point sets that Burges's own caveat excludes. The n+2 result (0/100, 0/60) confirms the universal claim: no sampled configuration, degenerate or not, was ever shattered, because that side of the corollary is a dimension-counting fact about every set of n+2 points, not a probabilistic one. Nothing about 2011-vs-1998 compute matters to either half: the corollary is decided by linear independence of position vectors, and a 2026 datacenter checks more configurations per second than a 1998 workstation without changing what any single configuration finds.
The second checkable claim is about Eq. (3) itself rather than about a general theorem it depends on: Burges reports that, at 95% confidence and a training sample of size l = 10,000, "the graph shows that for h/l > 0.37 (and for η = 0.05 and l = 10, 000), the VC confidence exceeds unity, and so for higher values the bound is guaranteed not tight." That 0.37 crossover is not asserted from a theorem — it is read off a numerically evaluated curve, Figure 3, and so it can be recomputed exactly rather than merely re-plotted. <run_code label="vc-confidence-curve"> import numpy as np l = 10000.0 eta = 0.05 def vc_confidence(h, l=l, eta=eta): val = (h*(np.log(2*l/h)+1) - np.log(eta/4)) / l return np.sqrt(val) hs = np.arange(1, 9000) conf = vc_confidence(hs) ratio = hs / l idx = np.argmin(np.abs(conf - 1.0)) print("h/l at VC confidence == 1:", ratio[idx], " (Burges states ~0.37)") for r in [0.05, 0.1, 0.2, 0.3, 0.37, 0.4, 0.5]: h = r*l print(f"h/l={r:.2f} VC-confidence={vc_confidence(h):.4f}") print("monotonic increasing over full range:", np.all(np.diff(conf) > 0)) </run_code> The rerun found the crossing at h/l = 0.3731, matching Burges's stated 0.37 to the two significant figures he gives, and confirmed strict monotonicity of the VC confidence in h over the entire range tested — the property Figure 3 was drawn to illustrate ("The VC confidence is a monotonic increasing function of h. This will be true for any value of l."). Both checks are arithmetic, not experimental in any sense that involves hardware limits: evaluating a closed-form square root at nine thousand points is sub-millisecond work on any machine built after about 1980, and the fact that Burges computed his version by hand-plotting a graph in 1998 while this rerun computed it as a NumPy array in under a second changes the cost of confirming the claim, not the claim's truth. What the two reruns together demonstrate is the chapter's recurring point from the other direction: where Volume I's obstructions were walls of computation that a later decade's hardware simply removed, the VC-dimension bounds derived in this section are not obstructions at all but exact, checkable mathematical relationships — the kind of result a 2026 datacenter reproduces in microseconds and a 1998 paper reproduces on paper, with the same answer either way.
Two objects have to be defined before either the phrase "maximum margin" or "kernel trick" means anything, and Vapnik's own restatement of the 1965 construction — reprinted in 1998 as "The Support Vector Method of Function Estimation" — gives both in the paper's own notation. Fix a training set of labeled vectors x with y ∈ {1, −1}. Two finite subsets, subset I where y = 1 and subset II where y = −1, are called separable by a hyperplane if there exists a unit vector φ and a constant c such that (x_i · φ) > c for x_i in I and (x_j · φ) < c for x_j in II — the paper's own opening of the construction: "We say that two finite subsets of vectors x from the training set" satisfying those inequalities. Among all such separating φ, define c₁(φ) as the minimum of (x·φ) over subset I and c₂(φ) as the maximum of (x·φ) over subset II, and let ρ(φ) = ½(c₁(φ) − c₂(φ)) be the margin that φ achieves. The paper picks out one φ from all the separating ones: "Consider the unit vector f/Jo which maximizes the function" ρ(φ) subject to the separation constraints — the Optimal hyperplane, or Maximal Margin hyperplane. A short convexity argument in the same section rules out ties: "The Optimal hyperplane is unique," proved by observing that ρ is achieved on the boundary of the unit ball and that two maximizing boundary points would force the maximum to recur, impossibly, at an interior point on the segment between them. Recast as a constrained minimization — find ψ₀ and threshold b₀ satisfying y_i((x_i·ψ₀) + b) ≥ 1 for every training pair such that "the vector 1/Jo has the smallest norm" — the optimal hyperplane becomes a quadratic program: minimize ‖ψ‖² subject to one linear inequality per training point. Nothing in this construction has yet mentioned a kernel, a feature space, or a nonlinear transformation; it is a statement about points in the original input space, and it says only that among the (possibly many) hyperplanes that separate two labeled clouds without error, exactly one maximizes the gap between them, and that hyperplane is what the rest of the chapter calls the support vector machine before any kernel is applied to it.
The route from that primal minimization to a computable algorithm goes through the dual, and it is the dual, not the primal, that everything downstream of this section depends on. Introduce one non-negative multiplier α_i per training constraint and form the Lagrangian; the paper states the standard machinery plainly — "one has to find the saddle point of the Lagrange function" L(ψ,b,α) = ½(ψ·ψ) − Σᵢ αᵢ(yᵢ[(xᵢ·ψ)+b] − 1), "where ai ;:::: 0 are the Lagrange multipliers," minimized over ψ and b and maximized over the αᵢ. Setting the partial derivatives to zero gives the two identities that carry the entire method forward: "from these conditions it follows that for the vector 'I/J that defines the Optimal hyperplane, the equalities" ψ = Σᵢ yᵢαᵢxᵢ and Σᵢ yᵢαᵢ = 0 hold. Substituting the first identity back into the Lagrangian eliminates ψ and b entirely, leaving a function of the multipliers alone, W(α) = Σᵢ αᵢ − ½ Σᵢⱼ yᵢyⱼαᵢαⱼ(xᵢ·xⱼ), to be maximized subject only to αᵢ ≥ 0 and Σᵢ yᵢαᵢ = 0 — a quadratic program whose only reference to the data is through the pairwise dot products xᵢ·xⱼ, arranged, as Cortes and Vapnik write it in their 1995 "Support-Vector Networks" paper, into a symmetric matrix Dᵢⱼ = yᵢyⱼ(xᵢ·xⱼ). Two consequences follow immediately, and both do real work later in the chapter. First, because the Karush-Kuhn-Tucker conditions force αᵢ(yᵢ[(xᵢ·ψ)+b] − 1) = 0 at the saddle point, any training vector for which the margin constraint holds with room to spare — strictly on the correct side of the margin — must have αᵢ = 0 and drops out of ψ = Σᵢ yᵢαᵢxᵢ entirely; only the vectors lying exactly on the margin, the support vectors, carry nonzero weight. Second, since the objective and the solution depend on the data only through inner products xᵢ·xⱼ, nothing in this derivation has used the coordinates of x individually — a fact whose consequences are the entire subject of the next block. The 1965 generalization bound that had sat unused for three decades before Boser, Guyon, and Vapnik's paper falls directly out of this same sparsity: since the classifier's parameters are a weighted sum of the support vectors alone, a test point's classification depends only on how many training points ended up as support vectors relative to the training set size, which is why Cortes and Vapnik's 1995 paper could state it directly — E[Pr(error)] is bounded by "the ratio between the expectation value of the number of support vectors and the number of training vectors," in language reused when this history discussed the paper's postal digit results in Chapter 6 — a bound that, as Cortes and Vapnik note, "does not explicitly contain the dimensionality of the space of separation."
The dual's dependence on data only through dot products is what makes the kernel trick possible, and Cortes and Vapnik's paper is explicit that the trick itself predates their soft-margin extension by three decades and was not their idea. The technical obstacle they name outright is dimensionality: to build a degree-four or degree-five polynomial decision surface over a mere two-hundred-dimensional input, the naive approach — map each x explicitly into its feature space Z of monomials, then take dot products there — would require constructing hyperplanes in a space of on the order of a billion dimensions. The move that avoids ever forming that space is a reordering, credited to Boser, Guyon, and Vapnik's 1992 paper: "it was shown (Boser, Guyon, & Vapnik, 1992), that the order of operations for constructing a decision function can be interchanged: instead of making a non-linear transformation of the input vectors followed by dot-products with support vectors in feature space, one can first compare two vectors in input space (by e.g. taking their dot-product or some distance measure), and then make a non-linear transformation of the value of the result." Write K(u,v) for this input-space comparison function; the dual quadratic program becomes W(α) = Σᵢαᵢ − ½Σᵢⱼ yᵢyⱼαᵢαⱼK(xᵢ,xⱼ), identical to the previous block's derivation with every xᵢ·xⱼ replaced by K(xᵢ,xⱼ), and the cost of evaluating K is whatever it costs to compare two vectors in the original, low-dimensional input space — independent of how large a feature space K implicitly represents. What licenses this substitution mathematically is a condition on K alone, not on any feature map: "Functions that satisfy Mercer's theorem can therefore be used as dot-products" — that is, any symmetric K for which ∬K(u,v)g(u)g(v) du dv > 0 for every square-integrable g corresponds to an actual inner product (Σᵢ λᵢφᵢ(u)φᵢ(v)) in some Hilbert space Z, whether or not anyone ever writes down the φᵢ. The paper is equally explicit that this move had already been made once before, in a different vocabulary, in the Soviet literature on pattern recognition: "Aizerman, Braverman and Rozonoer (1964) consider a convolution of the dot-product in the feature space" of the form K(u,v) = exp(−|u−v|²/σ²) — the function now called the Gaussian or radial basis kernel — "which they call Potential Functions." Aizerman, Braverman, and Rozonoer had the identity K = ⟨φ(u),φ(v)⟩ available in 1964; what they lacked was the 1965 optimal hyperplane to combine it with, since a potential function without a maximum-margin criterion behind it is only a similarity measure, not yet a classifier with a generalization guarantee attached. The kernel trick and the maximum margin are, on this reading, two separate results, arrived at three decades apart by mostly separate literatures, that turn out to compose without friction because both a margin-maximizing hyperplane and a Mercer kernel reduce a learning problem to nothing but pairwise comparisons between training points.
Put the two pieces together and the setup for the rest of this chapter is in place. Chapter 6 derived a bound on generalization error in terms of the VC dimension h of a hypothesis class, a bound Burges emphasizes is remarkable chiefly because "it is independent of P(x, y)" — it holds without any assumption about the distribution generating the data. The optimal hyperplane construction of this section supplies a hypothesis class — oriented hyperplanes with margin at least ρ — whose VC dimension can be bounded not by the dimensionality n of the space they live in but by a quantity governed by the margin itself and the radius of the smallest ball containing the data; a hyperplane that separates with a wide margin belongs to a small class in Burges's sense even when n is a billion. That is what makes the kernel trick more than a computational shortcut: mapping into a high- or infinite-dimensional feature space would be a capacity catastrophe for an ordinary linear classifier, whose VC dimension in R^n is n+1, but it is not a catastrophe for a margin classifier, whose effective capacity is controlled by ρ rather than n. The kernel trick of the previous block supplies the other half — a way to search that same high-dimensional space computationally, at the cost of comparing input vectors rather than constructing feature vectors, provided the comparison function satisfies Mercer's condition. Neither piece alone gives a usable algorithm: a maximum-margin hyperplane restricted to the original input coordinates is often just a linear classifier with a generalization bound and no way to fit nonlinear structure; a Mercer kernel without a margin criterion is a well-defined but ungrounded similarity measure, of exactly the kind Aizerman, Braverman, and Rozonoer had in 1964. It is the combination — choosing among hyperplanes in a kernel-induced feature space by "the essential idea of structural risk minimization," minimizing a bound that is a function of the margin and the empirical error rather than of the empirical error alone — that the next section derives as a single quadratic program, first for the separable case, then, following Cortes and Vapnik's soft-margin extension — already introduced narratively in this book's first volume — for data that no hyperplane in any feature space can separate without error at all.
The previous section's dual quadratic program presupposes that the two classes can be separated without error — that some φ achieves ρ(φ) > 0 — and Cortes and Vapnik's 1995 paper opens its central extension by naming exactly the case that assumption excludes: "Consider the case where the training data cannot be separated without error. In this case one may want to separate the training set with a minimal number of errors." Their fix keeps the primal minimization from section 1 almost unchanged — note only that Cortes and Vapnik's own 1995 paper writes the weight vector as w and the threshold as b, where the previous section, following Vapnik's 1998 restatement, wrote ψ and b; the two letters name the same object, and this section follows each paper's own usage rather than forcing one notation onto both. For each training pair, "let us introduce some non-negative variables" ξ_i ≥ 0, one per training vector, and relax the margin constraint from y_i(w·x_i + b) ≥ 1 to y_i(w·x_i + b) ≥ 1 − ξ_i, so that a point may now sit inside the margin, or even on the wrong side of the hyperplane, at the cost of a positive ξ_i. Minimizing Σξ_i alone would count training errors, but that combinatorial problem is NP-complete, so Cortes and Vapnik instead minimize the joint functional ½(w·w) + C·Σᵢ ξᵢ — their equation (24) with the penalty function F(u) = u — subject to the relaxed margin constraints and ξᵢ ≥ 0, where C is a constant fixed in advance that trades the width of the margin against the total slack incurred by misclassified or marginal points. Note what has not changed: the objective is still ½(w·w), the same quantity minimized in the separable case, and as C is taken larger the penalty on slack dominates and the solution approaches the hard-margin hyperplane of the previous section whenever one exists; the soft-margin construction is the hard-margin construction with one additional term, not a different algorithm.
The dual of this new primal is obtained by the identical route as before — form the Lagrangian, differentiate with respect to the primal variables w, b, and now also ξᵢ, and set each to zero. The stationarity condition with respect to w reproduces w = Σᵢ yᵢαᵢxᵢ exactly as in the hard-margin case, and the condition Σᵢ yᵢαᵢ = 0 also survives unchanged; what is new is the derivative with respect to ξᵢ, which — because ξᵢ appears in the objective with coefficient C and in the Lagrangian multiplied by a second, non-negative multiplier attached to the constraint ξᵢ ≥ 0 — forces that second multiplier to equal C − αᵢ, and since it too must stay non-negative, the dual variable αᵢ is capped: 0 ≤ αᵢ ≤ C. The dual objective W(α) itself is untouched — it is the same Σᵢαᵢ − ½Σᵢⱼyᵢyⱼαᵢαⱼ(xᵢ·xⱼ) as in the separable case — and maximizing it subject to Σᵢyᵢαᵢ = 0 and the box constraint 0 ≤ αᵢ ≤ C is, in Cortes and Vapnik's words, still "a quadratic programming problem," now with an upper bound added to the feasible region rather than a different feasible region altogether. The Karush-Kuhn-Tucker complementary-slackness conditions read off the solution's structure: a training vector with αᵢ = 0 lies outside the margin and is classified correctly with room to spare, exactly as before; a vector with 0 < αᵢ < C sits exactly on the margin and has ξᵢ = 0; and a vector that has been pushed inside the margin or misclassified — ξᵢ > 0 — must have its multiplier pinned at the upper bound, αᵢ = C. The support vectors are now these two populations together — the margin vectors and the bounded errors — and the same expansion, w = Σᵢyᵢαᵢxᵢ, sums over both. Because the box constraint and the equality constraint are the only two changes to the quadratic program, everything the previous section established about kernels transfers without modification: replacing every xᵢ·xⱼ in this dual by K(xᵢ,xⱼ) gives the soft-margin decision function in feature space, and the generalization bound tied to the count of support vectors continues to hold, except that the support vectors now include the points the margin was willing to sacrifice.
The margin construction generalizes a second way, beyond tolerating errors, to a task that is not classification at all. Vapnik's 1995 reformulation of the same quadratic-program machinery for regression — set out in Smola and Schölkopf's tutorial in the paper's own notation — asks for a function f(x) = ⟨w,x⟩ + b that stays within a fixed tolerance ε of every training target yᵢ while remaining as flat as possible: "we do not care about errors as long as they are less than ε, but will not accept any deviation larger than this." Flatness again means minimizing ‖w‖², now subject to two one-sided constraints per point, yᵢ − ⟨w,xᵢ⟩ − b ≤ ε and ⟨w,xᵢ⟩ + b − yᵢ ≤ ε, which together carve out an ε-wide tube around f and penalize nothing inside it. Exactly as Cortes and Vapnik's soft margin admits points that violate the hard constraint, slack variables ξᵢ, ξᵢ* are introduced for targets that fall outside the tube, giving the primal minimize ½‖w‖² + C·Σᵢ(ξᵢ + ξᵢ*) subject to yᵢ − ⟨w,xᵢ⟩ − b ≤ ε + ξᵢ, ⟨w,xᵢ⟩ + b − yᵢ ≤ ε + ξᵢ*, and ξᵢ, ξᵢ* ≥ 0 — the ε-insensitive loss |ξ|_ε, zero inside the tube and linear beyond it, playing the same role C's slack penalty played for misclassification. Introducing four multipliers per point (ηᵢ, ηᵢ*, αᵢ, αᵢ*, one pair for the slacks and one pair for the tube constraints) and setting the Lagrangian's partial derivatives to zero gives ∂_bL = Σᵢ(αᵢ*−αᵢ) = 0, ∂_wL = w − Σᵢ(αᵢ−αᵢ*)xᵢ = 0, and ∂_{ξᵢ}L = C − αᵢ − ηᵢ = 0, the last of which — since ηᵢ ≥ 0 — again forces the box constraint αᵢ, αᵢ* ∈ [0,C]. Substituting back eliminates w, b, and the slacks, leaving the dual: maximize −½Σᵢⱼ(αᵢ−αᵢ*)(αⱼ−αⱼ*)⟨xᵢ,xⱼ⟩ − εΣᵢ(αᵢ+αᵢ*) + Σᵢyᵢ(αᵢ−αᵢ*) subject to Σᵢ(αᵢ−αᵢ*) = 0 and αᵢ,αᵢ* ∈ [0,C], with w recovered as Σᵢ(αᵢ−αᵢ*)xᵢ and consequently f(x) = Σᵢ(αᵢ−αᵢ*)⟨xᵢ,x⟩ + b — the same support-vector expansion as classification, with the coefficient yᵢαᵢ of the classification dual replaced by the signed difference αᵢ−αᵢ*. The Karush-Kuhn-Tucker conditions again force sparsity: for any point strictly inside the tube, |f(xᵢ)−yᵢ| < ε, both multipliers must vanish, so only points on or outside the tube boundary — the support vectors of the regression — contribute to the expansion, which is exactly why the tutorial can note that "the complexity of a function's representation by SVs is independent of the dimensionality of the input space X, and depends only on the number of SVs." Replacing every ⟨xᵢ,xⱼ⟩ in the dual by a Mercer kernel k(xᵢ,xⱼ) — the identical substitution licensed in the previous section — turns the flat linear function in feature space into an arbitrary nonlinear regression function in input space, at the same computational cost as before: the size of the kernel matrix, not the dimensionality of the implicit feature space.
The same substitution — dot products replaced by a Mercer kernel — reaches even further than regression, into a setting with no labels at all. Schölkopf, Smola, and Müller's 1998 kernel PCA takes ordinary principal component analysis, which "diagonalizes the covariance matrix" C = (1/M)Σⱼxⱼxⱼᵀ of a centered sample x₁,...,x_M ∈ ℝᴺ by solving λv = Cv, and rewrites it so that the data enter only through dot products, exactly as the paper states its aim: "We are interested not in principal components in input space but in principal components of variables, or features, which are nonlinearly related to the input variables." The rewriting exploits the same fact used throughout this chapter: because Cv = (1/M)Σⱼ(xⱼ·v)xⱼ, every eigenvector v with λ ≠ 0 must lie in the span of the training points, so there exist coefficients α₁,...,α_M with v = Σᵢαᵢxᵢ, and the eigenvalue equation reduces to a set of M equations involving only the pairwise dot products xₖ·xᵢ. Map each xᵢ into a feature space F by Φ, form the feature-space covariance C̄ = (1/M)Σⱼ Φ(xⱼ)Φ(xⱼ)ᵀ, and the identical argument goes through with V = Σᵢαᵢ Φ(xᵢ): defining the kernel matrix Kᵢⱼ = (Φ(xᵢ)·Φ(xⱼ)) = k(xᵢ,xⱼ), the eigenvalue problem in the (possibly infinite-dimensional) feature space collapses to the finite-dimensional problem Mλα = Kα, an M×M eigenvalue problem in the kernel matrix alone, with the normalization λₖ(αᵏ·αᵏ) = 1 fixing each eigenvector's length in F. Projecting a test point x onto the k-th feature-space eigenvector then costs nothing beyond evaluating the kernel against every training point: (Vᵏ·Φ(x)) = Σᵢαᵢᵏ k(xᵢ,x). No labels, no margin, and no slack variables appear anywhere in this derivation — the object being decomposed is the covariance of the data, not a separating hyperplane — and yet the computational shape is the one this section has now derived three times: form a kernel (Gram) matrix from pairwise comparisons of the training points, solve a matrix eigenvalue or quadratic-programming problem whose size is the sample count rather than the feature dimension, and express the solution — support vectors in classification and regression, principal directions in kernel PCA — as a weighted sum over training points that is evaluated on new data by further kernel comparisons. What Cortes and Vapnik's slack variables did for margins that could not be perfect, and Vapnik's ε-tube did for functions that could not be flat, kernel PCA does for a covariance structure that was never linear to begin with: each keeps the original quadratic or eigenvalue machinery intact and relaxes or repurposes only the object it is applied to, which is why a single Mercer kernel — one function satisfying one positivity condition — underwrites classification, regression, and unsupervised feature extraction without three separate derivations of what a kernel is permitted to be.
Cortes and Vapnik's 1995 paper reports, in its own words, "two types of experiments": an artificial 2D demonstration in Section 6.1 — decision surfaces fit to hand-constructed planar patterns, unrelated to digit recognition and not rerun here — and the digit-recognition work of Section 6.2, which splits in turn into the USPS and NIST databases. The chapter's derivations above map directly onto that digit-recognition pair: the hard-margin quadratic program of Section 1 is what runs when a polynomial kernel happens to separate the training set without error, and the soft-margin extension of Section 2 is what runs when it does not. Their stated plan was explicit about what the experiments were for: "In Section 5 we will demonstrate how well it generalizes for high degree polynomial decision surfaces (up to order 7) in a high dimensional space (dimension 256)." The USPS experiment used the "small" US Postal Service database — 7,300 training patterns and 2,000 test patterns of 16×16 pixel digits — and swept the polynomial kernel K(u,v) = (u·v+1)^d over degrees 1 through 7, tracking three quantities at once for each degree: the test error rate, the number of support vectors, and the dimensionality of the implicit feature space the kernel was standing in for, which for degree 7 the paper puts at roughly 10^16 — larger than the number of atoms in a human body, reached without ever constructing a single feature vector. Their Table 2 shows raw error falling from 12.0% at degree 1 (an ordinary linear classifier) to 4.7% at degree 2 and then wobbling in a narrow band, 4.2–4.4%, for degrees 3 through 7, while the support-vector count runs 200, 127, 148, 165, 175, 185, 190 across degrees 1 through 7 — a sharp drop from the linear classifier to the degree-2 polynomial, followed by only a slow partial climb back up that never regains the degree-1 count. That is the empirical content behind the paper's own comment on the table, "Notice that the number of support vectors increases very slowly," and behind its more pointed remark a sentence later: "The 7 degree polynomial has only 30% more support vectors than the 3rd degree polynomial — and even less than the first degree polynomial." The dimensionality of the implicit feature space, by contrast, does climb monotonically across the same seven rows, from 256 to roughly 10^16 — the comparison the table is built to make is between that runaway growth in nominal dimensionality and the near-flat, non-monotonic support-vector count, not a rise in the support-vector count itself. The NIST experiment scaled up to a database of 60,000 training and 10,000 test patterns and constructed only a single, 4th-degree polynomial classifier — the paper notes that "the training time for construction of polynomial classifiers does not depend on the degree of the polynomial" but only on the number of support vectors — and reports that "the combined performance of the ten classifiers on the test set is 1.1% error," a figure the paper sets directly against a linear classifier, a 3-nearest-neighbor classifier with 60,000 prototypes, and the two purpose-built LeNet1 and LeNet4 networks from the same 1994 benchmark study.
Rerunning the first experiment exactly is not possible — the 1995 USPS database of 7,300 training and 2,000 test digit bitmaps is not part of the offline corpus this book draws on — so the rerun below substitutes scikit-learn's bundled digits set: 1,797 handwritten-digit images, 8×8 pixels (64 dimensions rather than the paper's 256), split 1,257/540 train/test. This is a smaller, cleaner, lower-resolution proxy for the paper's "small" database, and every number below should be read with that substitution in mind, not as a literal reproduction. Training an SVC with a polynomial kernel K(u,v) = (u·v+1)^d (gamma=1, coef0=1, C=10, scikit-learn's libsvm-based solver rather than Cortes and Vapnik's own chunking QP code) at degrees 1 through 7 gave: degree 1, 1.85% test error, 371 support vectors; degree 2, 0.74%, 422; degree 3, 0.74%, 412; degree 4, 0.74%, 394; degree 5, 0.56%, 370; degree 6, 1.30%, 348; degree 7, 1.48%, 324. The central qualitative claim replicates: the largest single gain in accuracy comes from moving off the linear classifier (degree 1) to degree 2, after which error is flat to within a point across degrees 3 through 5, exactly the "almost does not change with increasing dimensionality" pattern Table 2 reports for USPS, even though the feature space here also explodes combinatorially with degree — the degree-7 polynomial feature space has on the order of 10^9 dimensions for this rerun's 64-pixel input, by the exact count of monomials of degree at most 7 in 64 variables (C(71,7) ≈ 1.33×10^9). The equivalent exact count for the paper's 256-pixel input is C(263,7) ≈ 1.59×10^13 — a four-order-of-magnitude jump from the rerun's own figure, tracking the four-fold difference in raw input dimensionality — but that exact count is itself three orders of magnitude smaller than the ~10^16 Cortes and Vapnik report in Table 2 for degree 7, and the two numbers should not be read as the same quantity. That discrepancy is not a rerun artifact: checking the exact combinatorial count against every row of Table 2 shows the paper's own figures tracking the cruder estimate n^d (256^7 ≈ 7.2×10^16, 256^6 ≈ 2.8×10^14, 256^5 ≈ 1.1×10^12) far more closely than the exact coefficient count, especially from degree 4 up, which suggests Table 2 reports an order-of-magnitude estimate of the feature space's size rather than the exact monomial count used here and in this rerun's own 64-dimensional figure. One thing diverges sharply from Table 2, and that divergence is the finding, not an error to smooth over: the rerun's absolute error is far lower at every degree (1.85% vs. the paper's 12.0% at degree 1, 0.56–0.74% vs. its 4.2–4.7% plateau) — expected, since 8×8 scikit-learn digits are a cleaner, lower-variance, single-source dataset, not scanned mail. On support vectors the two runs do not share a direction at all — they mirror each other. Cortes and Vapnik's own count drops sharply from degree 1 to degree 2 (200 to 127) and then climbs slowly and unevenly through degree 7 (148, 165, 175, 185, 190), ending well below its degree-1 value but well above its degree-2 minimum: a sharp drop followed by only a slow partial climb back up, as already noted above. This rerun's count does the opposite: it rises from degree 1 to degree 2 (371 to 422) and then declines monotonically through degree 7 (412, 394, 370, 348, 324) — a climb followed by a steady fall, the reverse shape from the paper's late-degree recovery. What Table 2 does not show, and this rerun does, is that decline's interaction with error: the paper's error is flat within 0.2 points from degree 3 on (4.4, 4.3, 4.3, 4.2, 4.3 across degrees 3–7 in Table 2) regardless of its support-vector count creeping back up, while in this rerun the monotonic post-degree-2 decline in support vectors coincides with error staying flat only through degree 5 and then rising — from 0.56% at degree 5 to 1.30% at degree 6 and 1.48% at degree 7 — exactly where the support-vector count is falling fastest (370 to 348 to 324). That combination — fewer support vectors and worse error at the same time, at the highest degrees — is a mild sign of the higher-order polynomial kernel beginning to overfit a training set (1,257 patterns) nearly six times smaller than USPS's 7,300, where the same nominal feature-space explosion buys less protection because there is less data to protect. Cortes and Vapnik do not themselves offer a mechanism for the shrinking-then-slowly-climbing support-vector count; their comment on it is descriptive, not explanatory — they report only that "the number of support vectors increases very slowly" and that performance "almost does not change with increasing dimensionality of the space — indicating no over-fitting problems." One plausible reading of that near-flat count, offered here as this book's inference rather than the paper's own claim, is that a good decision rule on this kind of data does not need many more "borderline" points as the space grows; that reading fits the degree-1-to-2 drop but does not obviously account for the climb from degree 2 to degree 7 that follows. On the smaller rerun dataset, by contrast, the count keeps shrinking well past degree 2, and that continued shrinkage is what coincides with rising rather than flat error. The two datasets diverge here rather than agreeing in direction, and that divergence — not a shared monotonic decline — is what this rerun adds to Table 2.
The second experiment — a 4th-degree polynomial classifier trained on the full 60,000-pattern NIST set — is the one whose scale this book's rerun can actually probe, because the bottleneck Cortes and Vapnik describe is not primarily "our machine was slow" but "the quadratic programming problem does not fit in memory at this size": the matrix D in equation (15) has one entry per pair of training vectors, so 60,000 patterns means a matrix of 3.6 billion entries, and their answer was an algorithm, not a bigger machine — "Divide the training data into a number of portions with a reasonable small number of training vectors in each portion," solve each portion's QP, keep only the support vectors, fold in the next portion, and repeat, so that "The training time for construction of polynomial classifiers does not depend on the de- gree of the polynomial---only the number of support vectors" but only on how many support vectors accumulate. Rerunning a degree-4 polynomial kernel on MNIST (784-dimensional, a modern stand-in for the paper's 28×28 NIST digits) at growing training-set sizes, using scikit-learn's libsvm solver with no chunking of our own, gives: 1,000 patterns, 0.08 seconds, 13.80% test error, 575 support vectors; 2,000 patterns, 0.19 seconds, 11.45%, 914; 4,000 patterns, 0.55 seconds, 8.70%, 1,413; 8,000 patterns, 1.65 seconds, 6.55%, 2,173; 16,000 patterns, 6.05 seconds, 4.60%, 3,371; 30,000 patterns, 21.29 seconds, 3.40%, 4,863. Two things are visible in that table at once. The error is converging toward the region of Cortes and Vapnik's own reported numbers — their combined ten-classifier result at the full 60,000 was 1.1% error, and this rerun's error is still falling at 30,000 with no sign of having found a floor. But the fit time is not falling; it is growing worse than linearly — roughly quadrupling from 8,000 to 30,000 patterns (a 3.75× increase in data, a 13× increase in time) — which is the dual QP's O(l^2)-to-O(l^3) cost surfacing directly, on hardware Cortes and Vapnik did not have.
Pushing the rerun to the paper's own scale settles the question this chapter keeps asking. Training the same unmodified, un-chunked degree-4 polynomial SVM on the full 60,000 MNIST patterns — the size Cortes and Vapnik needed their portion-by-portion QP algorithm to reach at all in 1995 — took 83 seconds on ordinary 2020s laptop hardware, for 2.70% test error and 7,254 support vectors (lab run 130; an earlier run, lab run 117, logged 664.44 seconds of wall-clock time, but roughly 581 of those seconds (664.44 minus the 83.30-second fit_time actually measured in run 130) were the machine asleep, not computing — the corrected, sleep-free figure is the one used throughout this section). Two different answers to "would it still bind with a 2026 datacentre" are visible in that single number, and they should not be collapsed into one. The specific ceiling the paper hit — a 3.6-billion-entry kernel matrix that would not fit in a 1995 workstation's memory, forcing a bespoke out-of-core algorithm before 60,000 patterns could be trained on at all — is simply gone; this rerun used no chunking, no portioning, nothing Cortes and Vapnik had to invent, and it still ran to completion on a single machine one person owns. But the deeper reason chunking was necessary in the first place has not gone anywhere: the jump from 30,000 to 60,000 patterns doubled the data and multiplied the fit time by roughly 3.9× (21.29s to 83.30s, matching the 30,000-pattern timing already reported above), the fingerprint of a dual quadratic program whose cost grows worse than linearly, closer to quadratically than linearly, in sample count — a property of the optimization problem in equations (15)–(17), not of any particular chip. That is why the fifteen years between this paper and the book's 2011 cutoff were not spent waiting for Moore's Law to make exact kernel SVMs cheap at web scale; they were spent on Platt's SMO (1998), which replaced chunking's block QP solves with pairwise updates cheap enough to avoid a QP library altogether, and later on low-rank and random-feature approximations that gave up exactness on purpose to buy back the linear scaling the dual problem does not have. The rerun's numbers confirm Cortes and Vapnik's central claim — a fixed-degree polynomial kernel keeps improving as data grows, with no sign of saturating even past 30,000 patterns — while confirming, on today's hardware, that the cost of computing that improvement exactly is still not a hardware problem with a hardware answer.
Start from the fact that a joint distribution over n discrete variables, written out with no structure at all, needs one number for every combination of their values — 2^n entries if each variable is binary — and grows so fast that no domain of any size can have its joint distribution simply tabulated, elicited from an expert, or estimated from data. The Bayesian network is a proposal for representing that same joint distribution using far fewer numbers, by fixing in advance which variables are allowed to depend on which others and forcing every other dependency to be absent. The representation is a directed acyclic graph, one node per variable, together with one probability table per node — a prior P(x) for a node with no parents, a conditional P(x|parents) for every other node — and the central claim of the formalism, proved rather than assumed, is that these local tables determine the entire joint distribution. Cooper's 1990 analysis states the result directly: "Therefore, the joint probability of any instantiation of all the variables in a belief network can be computed as the product of only n probabilities." Nothing about that claim is true of an arbitrary collection of conditional probability tables; it holds only because the graph's missing edges are read as an assertion of conditional independence, and it is that assertion — not the graph's causal-sounding arrows — that does the mathematical work. As Cooper's 1989 survey puts it, "The key feature of belief networks is their explicit representation of the conditional independence among the events (variables) in rules," and the rest of this section makes precise what that representation actually asserts, before the next section derives what can be computed from it.
Cooper's 1989 survey makes the independence assumption exact using a small five-node network — metastatic cancer (A) causing both increased serum calcium (B) and a brain tumour (C), each of which can cause coma (D), with the tumour additionally causing papilledema (E) — worked with actual numbers: P(A=T)=0.001, P(B=T|A=T)=0.3, P(C=T|A=T)=0.2, and a table of P(D|B,C) with four entries. The graph asserts something the numbers alone do not: "Belief-network semantics require that increased total serum calcium and brain fumour be conditionally independent given metastatic cancer" — B and C, both effects of the same cause A, do not inform each other once A is known, and the missing edge between them is precisely that claim. Cooper states the general version by defining, for each node x_i, its direct predecessors π_i and the set Ω_i of subsets of its non-descendants; the network asserts that for every such subset φ, P(x_i | π_i ∪ φ) = P(x_i | π_i) — "This relation states that if the values of the direct predecessors of x; are known with certainty, then the probability of each value of x; is conditionally independent of any subset of the nodes in Si." Applied to the coma node D, whose parents are B and C: "In the previous example, knowing the values of nodes B and C is sufficient to determine the probability that node D = T, regardless of the values of nodes A and E" — the cause of the causes, and the further effect of one of them, both drop out once the immediate parents are fixed. The graph's edges are read the same way everywhere: "The absence of an arc from a node Xj in Si to node Xi indicates that X; is conditionally independent of xj, given the values of the nodes in Ti" — every missing arrow is a conditional-independence statement, and a Bayesian network is nothing more than a compact way of listing which ones hold, node by node, so that the product of the resulting local tables reconstructs the full distribution over all five variables from eleven numbers instead of the thirty-one a five-variable joint would otherwise require.
The general DAG formalism did not arrive fully formed; Pearl's 1986 note on evidential reasoning in a hierarchy of hypotheses works out the same conditional-independence machinery for the restricted case of a tree, where each hypothesis has exactly one parent and evidence is propagated up toward the root and back down toward the leaves by local, node-to-neighbour messages rather than by any global recomputation. Pearl frames the problem explicitly as an extension of an existing scheme for a more general representation: "The purpose of this note is to supplement the G-S analysis with a description of how evidential reasoning can be conducted in the same hypothesis space using a Bayesian formalism." The updating rule he derives — redistribute belief across siblings in proportion to their prior weights and the likelihood of the evidence, then normalize — depends on nothing about trees specifically once it is stated correctly in conditional-independence terms, and Pearl says so directly, in the sentence that is the real hinge between the tree-structured precursors and the general belief network: "Note that an identical weight-distribution process, followed by normalization, would still be valid in a general graph hierarchy, not just trees." What changes between a tree and an arbitrary DAG is not the validity of local, neighbour-to-neighbour updating in principle — that survives — but the ease of guaranteeing it converges to the right answer, since a tree gives each node exactly one path to any other node and a general DAG does not. The rest of this chapter's setup is the work of making that one sentence precise: stating exactly which conditional-independence assumptions a general DAG encodes, and exactly what has to be added — the moralization step below, and the junction-tree construction of the next section — to make local propagation exact once the single-path guarantee is gone.
Lauritzen and Spiegelhalter's 1988 paper draws on two examples, and it matters which is which. The product formula Cooper proves abstractly is stated concretely against MUNIN, a real expert system for muscle-and-nerve (EMG) diagnosis built at Aalborg, whose Fig. 1 "shows a 25-node representation of a single muscle" with an 11-state disease node: "Having specified this conditional probability table for each node, we then make the crucial assumption (Kiiveri et al., 1984) that our joint probability for a particular set of 25 states equals the product over the entries in the 25 conditional probability tables that feature the appropriate states." Their qualitative reading of the same assumption is the Markov property that gives local computation its name: knowledge of a node's parents screens off everything else except its descendants — "this represents our judgement that if we know the values of parents of a node v whose value is currently unknown, then no other knowledge (except concerning descendants of v) will influence" one's belief about that node. But MUNIN is too large to step through by hand, so the paper switches to a second, much smaller network for its worked illustration: "a small piece of fictitious qualitative medical 'knowledge'" about shortness of breath (dyspnoea), a recent visit to Asia, tuberculosis, lung cancer, bronchitis and smoking — the chest-clinic example. It is here that the paper shows why the DAG itself is the wrong graph to compute on directly: a node with two or more parents ties those parents together in its conditional table without there being an edge between them in the original directed graph; the fix is to add one, joining every pair of parents that share a child and then discarding the arrowheads entirely. "We call this the moral graph (formed by 'marrying' parents)," and the reason the construction is forced rather than cosmetic is stated just as plainly: "(4.3) only involves functions on sets of nodes which are complete (all nodes joined) subgraphs of the moral graph" — the local tables only ever multiply together over cliques of the moral graph, never over sets that the directed graph alone would suggest, so it is the moral graph, not the DAG, that determines which computations can stay local.
That marriage of parents is also where the difficulty the next section has to confront first becomes visible: Cooper's 1990 hardness result, proved by reduction from 3SAT on precisely the multiply-connected networks that moralization tends to produce, shows the difficulty is not a matter of patience or hardware. Cooper states it without qualification: "we show that probabilistic inference using multiply connected networks is NP-hard," and draws the negative conclusion immediately: "it is unlikely that a general, efficient probabilistic inference algorithm can be developed for belief networks." The hard case cannot simply be defined away as a pathology of toy examples, either — Cooper notes beforehand that "it appears that large multiply connected networks are needed for some complex domains, such as medicine." What the moral graph buys, and what the next section spends, is a graph-theoretic route around that worst case: triangulate the moral graph, organize its cliques into a junction tree, and confine the exponential cost to the width of that tree rather than to the number of variables — belief propagation and the variational methods that follow it are, at bottom, different answers to the question of what to do when triangulation still leaves that width too large to enumerate.
Kschischang, Frey, and Loeliger's 2001 factor-graph paper generalizes the junction-tree machinery of the previous section by first stripping it of everything specific to probability. A factor graph has two kinds of nodes — one per variable, one per local factor — with an edge joining a variable to every factor that mentions it; the global function is the product of the local ones, exactly as the moral graph's cliques multiply together in the previous section, but now the "product" and the "sum" used to marginalize a variable out are allowed to be any pair of operations satisfying a distributive law: "the codomain of the global function... may in general be any semiring with two operations that satisfy the distributive law." Ordinary probability is the semiring of real multiplication and addition; substituting the "max-product" semiring, where summation becomes maximization, turns the identical message-passing machinery into the Viterbi algorithm's search for the single most probable configuration rather than a marginal over all of them; substituting "min-sum" in the negative-log domain gives the same result with additions instead of products. The point of the generalization is that the case for exactness is a fact about trees, not about probability: whenever the factor graph is cycle-free, clustering nodes together to remove any remaining cycle only ever changes which local computation is performed, never whether the answer is correct, and the paper states this as the general license for the whole family of exact inference algorithms — "clustering nodes may eliminate cycles in the graph so that the sum-product algorithm in the new graph computes marginal functions exactly." The generic algorithm this licenses is called sum-product or belief propagation: each node collects incoming messages from all its neighbours but one, combines them by the semiring's product, summarizes over its own variable by the semiring's sum, and sends the result onward; on a tree this process has a natural end, stated precisely in terms of message scheduling — "in a cycle-free factor graph, assuming a schedule in which only pending messages are transmitted, the sum-product algorithm will eventually halt in a state with no messages pending" — and the values sitting on the edges when it halts are exact marginals, computed in time linear in the graph rather than exponential in the number of variables. Junction trees, in this reading, are simply factor graphs whose variable nodes have been clustered until every remaining cycle disappears; sum-product run directly on the clusters is the same algorithm the previous section reached by triangulation, restated in a form that no longer depends on the factors being probability tables at all.
The paper's cleanest demonstration that sum-product is a genuine generalization, and not just a relabeling of the junction-tree algorithm for discrete variables, is its derivation of the Kalman filter as a single instance of the same rule applied to continuous Gaussian factors. The dynamical system is written x_t = a_t x_{t-1} + b_t u_t, y_t = c_t x_t + d_t w_t, with u_t and w_t independent unit-variance Gaussian noise and x_0 = 0; because linear combinations of jointly Gaussian variables are Gaussian, the whole state sequence and observation sequence are jointly Gaussian, and the factor graph's chain structure lets the joint density of the states given the observations be written, as in the discrete case, as a product of local Gaussian factors — "the Markov structure of this system permits us to write the conditional joint probability density function of the state variables... where [each factor] is a Gaussian density." The paper names the message passed forward along the chain m_{x_{t-1}→x_t}, notes it is "always of the form N(x; ...), and so may be represented by the pair" of a mean and a variance, and interprets that mean as "the MMSE prediction of x_t given the set of observations up to time t-1" — the summary operation here is not summation over a finite domain but integration over a continuum, the semiring generalization doing its stated work. Applying the sum-product update to this factor graph, "by completing the square in the exponent," yields a recursion for the pair (mean, variance) with one term the paper calls the "filter gain" — precisely the Kalman gain — and the paper draws the general moral explicitly: "these updates are those used by a Kalman filter... We note that similar updates would apply to any cycle-free factor graph in which all distributions (factors) are Gaussian. The operation of the sum-product algorithm in such a graph can, therefore, be regarded as a generalized Kalman filter, and in a graph with cycles as an iterative approximation to the Kalman filter." That last clause is the hinge to what follows: the same sentence that certifies exactness on a chain already names, in passing, the case where the chain grows cycles and the guarantee disappears.
Section V of the paper takes that hinge and pushes on it. Nothing in the sum-product update rule requires a tree — each node's outgoing message is computed from its local factor and its most recently received incoming messages regardless of the graph's global shape — so "the sum-product algorithm may also be applied to factor graphs with cycles simply by following the same message propagation rules, since all updates are local." What is lost is the termination argument of the previous block: "in a factor graph with cycles" it "is impossible to reach a state with no messages pending, since the transmission of a message on any edge of a cycle" from a node "will trigger a chain of pending messages that must return" to it, triggering it "to send another message on the same edge, and so on indefinitely." Running the algorithm anyway — picking a finite schedule and stopping — no longer computes a marginal in any provable sense: "in contrast with the cycle-free case, the results of the sum-product algorithm operating in a factor graph with cycles cannot in general be interpreted as exact function summaries." And yet the paper's next sentence is the one that made this section famous rather than merely honest: it names "the decoding of turbo codes or LDPC codes" as among the algorithm's most exciting applications, precisely because these "arise precisely in situations in which the" factor "graph does have cycles." A turbo code's parallel concatenated convolutional encoders, joined by an interleaver, produce a factor graph with exactly this structure, decoded by a schedule of forward-backward passes over one constituent code, exchange of "extrinsic information" with the other, and repetition; an LDPC code's Tanner graph is cyclic by construction. Both are decoded today by "flooding" every edge of the cyclic graph with sum-product messages for a fixed number of rounds and reading off the marginals as if the graph were a tree, with no guarantee that the result is even close to the true marginal — and the empirical outcome is stated without hedging: "with very long codes such decoding algorithms can achieve astonishing performance (within a small fraction of a decibel of the Shannon limit on a Gaussian channel) even though the underlying factor graph has cycles," while LDPC decoding under the same iterative rule is reported to reach "excellent performance results approaching that of turbo codes." This is worth sitting with rather than resolving too quickly, because it is exactly the kind of gap the book's running test is built to catch — and this one does not resolve the way the hardware limitations elsewhere in the corpus resolve. A 2026 datacentre changes nothing about whether loopy sum-product converges to the right marginal on an arbitrary cyclic graph: that is a question about a fixed-point iteration on a non-tree graph, not about clock speed, and the paper itself, writing in 2001 with the codes already in commercial deployment, offers no proof that the good behavior generalizes — only the empirical fact that on the graphs practitioners cared about, it did. The mathematics of exactness belongs to trees; the algorithm that was derived for trees was run somewhere else, and it worked, and explaining why remained — as far as this paper is concerned — open.
There is a second way to give up on exactness, and Beal and Ghahramani's variational Bayesian framework — building on Attias's original formulation — takes it. Where loopy sum-product keeps the exact local update rule and runs it on a graph the derivation never licensed, variational inference keeps the graph honest and instead restricts the space of candidate answers: it picks a family of distributions simple enough to optimize over, and asks for the closest member of that family to the true posterior, closest being measured by a bound built from a different device than the one behind sum-product: where the previous two blocks got exactness from the distributive law of an abstract semiring, applied to trees, variational inference gets its bound from Jensen's inequality applied to the concave logarithm — two distinct mathematical devices, not one reused. Writing y for the observed data, x for the hidden variables, θ for the model's parameters, and m for the model itself, the marginal likelihood the modeler actually wants is ln p(y | m), and introducing any distribution q(x, θ) with support wherever the true joint posterior has support gives, by Jensen, a lower bound: ln p(y|m) ≥ ∫dθ dx q(x,θ) ln[p(x,y,θ|m)/q(x,θ)]. Setting q(x, θ) equal to the exact posterior p(x, θ | y, m) turns this inequality into an equality — but that move is circular, since "evaluating the exact posterior distribution p(x, θ | y, m) requires knowing its normalising constant, the marginal likelihood," which is the very quantity out of reach. The escape is to give up on exactness deliberately rather than by accident: constrain q to the factorized form q(x, θ) = q_x(x) q_θ(θ), decoupling the hidden variables from the parameters even though the true posterior couples them, because, in the paper's words, "we use the variational methodology to instead work in the space of simpler distributions — those that are factorised between hidden variables and parameters." Substituting this factorized q back into the bound yields a functional F_m(q_x, q_θ) that can be maximized by coordinate ascent: a VBE step that updates q_x holding q_θ fixed, and a VBM step that updates q_θ holding q_x fixed, each one a direct generalization of the E and M steps of ordinary EM, with the paper affirming the same monotone guarantee EM enjoys — "each application of the VBE and VBM steps is guaranteed to increase or leave unchanged the lower bound on the marginal likelihood, and successive applications are guaranteed to converge to a local maximum." Collapsing q_θ to a point mass, a Dirac delta at a single parameter setting, erases the distinction entirely: the paper notes the algorithm then "reduces to the ordinary EM algorithm for ML" estimation, with the M step simply re-estimating that point. Variational Bayes is EM's generalization in exactly the direction Kalman filtering generalized tree-shaped sum-product in the previous block — replace a point estimate with a distribution — but applied to the parameters rather than the states. What this buys, and what it costs, both fall out of one identity. Rearranging the bound shows that the gap between the true log marginal likelihood and the variational bound F_m is precisely the Kullback-Leibler divergence between the factorized q_x(x) q_θ(θ) and the true joint posterior p(x, θ | y, m) — a quantity that is always non-negative and is zero only when the factorization happens to be exact. That gap is not a stand-in for missing computation; it is the price of an assumption, paid whether or not a faster machine is available to pay it with. A 2026 datacentre changes nothing about whether hidden variables and parameters are independent in the true posterior of a given model — that is a fact about the model, fixed the moment the model is written down — so the same factorized family will carry the same irreducible KL gap however many teraflops are thrown at optimizing within it. This is worth setting directly against loopy sum-product's failure mode from the previous block: loopy BP keeps a correct algorithm and runs it somewhere its correctness proof does not reach, discovering empirically that the answer is often good anyway; variational inference keeps a bound that is provably correct in one direction — it never overestimates the marginal likelihood — and pays for that guarantee with a gap whose size is fixed by a modeling choice made in advance. Both are responses to the same underlying fact, that exact inference in factor graphs and directed graphical models with hidden variables is exponential in the number of latent configurations regardless of clock speed; where they differ is that one gambles on an uncontrolled approximation and the other accepts a controlled, quantifiable one.
Section 8.1 introduced the "chest clinic" network qualitatively — visit to Asia, smoking, tuberculosis, lung cancer, bronchitis, the derived node "tuberculosis or cancer," X-ray, and dyspnoea — as an illustration of moralization and the junction tree. What that section left out is that Lauritzen and Spiegelhalter do not leave the example qualitative: their Table 1 assigns an actual number to every entry, closing the gap between "the graph asserts an independence" and "here is a joint distribution a machine can compute with." The paper is explicit that this is a toy assignment rather than an elicited clinical fact — "Fourteen probabilities require assessment, showing 'visit to Asia' as a fivefold risk factor for tuberculosis, 'smoking' as tenfold risk for lung cancer" — but a toy assignment with real numbers is exactly what a rerun needs: a prior P(visit to Asia = yes) = .01, P(smoker = yes) = .50, conditionals P(tuberculosis|Asia) of .05 against a background rate of .01, P(lung cancer|smoker) of .10 against .01, P(bronchitis|smoker) of .60 against .30, a deterministic OR gate for "tuberculosis or cancer," P(positive X-ray | either) of .98 against .05, and P(dyspnoea | either, bronchitis) ranging from .90 down to .10 across the four parent combinations. This section rebuilds that eight-node network from those fourteen numbers, computes the posterior the paper computed by hand with the junction-tree algorithm, and then asks the question the previous section's discussion of loopy propagation left open: run the same evidence through ordinary sum-product on the network's factor graph, which contains a cycle the junction-tree construction was built specifically to avoid, and see whether it converges, and to what.
The paper's own worked case history — reached via their junction-tree propagation by hand — conditions on a patient who has recently visited Asia, presents with dyspnoea, and receives a negative chest X-ray, and asks for the revised belief in bronchitis; their Fig. 8 commentary reports that "the circumstantial evidence for tuberculosis has been overwhelmed by the negative X-ray, leaving bronchitis as the only serious contender," and gives the number directly: "a revised probability p(bIa,d, x) = 0.863." A rerun does not need a junction tree at all for a network this size — eight binary variables have only 256 joint configurations, small enough to enumerate directly in software rather than route through cliques — so the check is a brute-force marginalization: fix the CPTs exactly as printed in Table 1, multiply out the joint probability of every configuration of (smoker, tuberculosis, lung cancer, bronchitis, "either") consistent with Asia=yes, dyspnoea=yes, X-ray=no, sum over everything except bronchitis, and normalize. Run in Python, this enumeration gives P(bronchitis = yes | evidence) = 0.8628 against the paper's reported 0.863 — a match to the third decimal place, the residual thousandth almost certainly the rounding the 1988 authors carried through their by-hand clique-marginal arithmetic rather than any disagreement about what the model implies. The same brute-force sweep gives the marginals the paper does not print a number for: P(smoker=yes | evidence) = 0.6045, P(tuberculosis=yes | evidence) = 0.0022, P(lung cancer=yes | evidence) = 0.0026, P("either"=yes | evidence) = 0.0047 — all small, consistent with the paper's own qualitative reading that a negative X-ray "overwhelmed" the circumstantial case for a Asia-linked tuberculosis diagnosis and left bronchitis, driven by the still-uncertain smoking history, as the standing explanation for the dyspnoea.
The interesting case is not the tree-shaped part of this network but the part that is not a tree. Absorb the evidence a=1 (visit to Asia), x=0 (negative X-ray), and d=1 (dyspnoea) into unary factors on their neighbors, and look at the factor graph connecting the five remaining variables {smoker, tuberculosis, lung-cancer, either, bronchitis}. It contains exactly one cycle: smoker connects to lung-cancer through fL, lung-cancer connects to either through fE, either connects to bronchitis through fD, and bronchitis connects back to smoker through fB. Kschischang, Frey, and Loeliger's sum-product algorithm is derived and proved correct by induction on message-passing schedules that terminate — the argument of their Section IV assumes the graph is a tree, and their Section V — titled "Iterative Processing: The Sum-Product Algorithm in Factor Graphs with Cycles" — is explicit that running the same local update rule on a graph with a cycle is only an iterative heuristic with "no natural termination," what the later literature (Murphy, Weiss, and Jordan among others) would christen "loopy" belief propagation; Kschischang, Frey, and Loeliger themselves never use that word. The junction-tree machinery of Lauritzen and Spiegelhalter's Section 4 exists precisely to avoid this: it triangulates the moral graph so the resulting clique tree has no cycles left to propagate around. The rerun asks the question directly — initialize every message on this five-node loopy graph to the uniform [0.5, 0.5], iterate the ordinary sum-product update with no scheduling cleverness, and watch what the belief at bronchitis does round by round. It does not sit at a fixed point: iteration 1 gives 0.6349, iteration 2 overshoots to 0.8294, iteration 3 overshoots past the true value to 0.8659, and only from iteration 4 onward does it settle — 0.8626, 0.8625, 0.8624, and then unchanged to four decimal places through iteration 20. The converged loopy value, 0.8624, sits 0.0004 below the brute-force exact marginal of 0.8628, and both sit within rounding of the paper's hand-computed 0.863.
Extending the same thirty-round loopy sweep to all five unobserved variables confirms the single converged case was not a fluke: smoker converges to 0.6044 against an exact 0.6045, tuberculosis to 0.0023 against 0.0022, lung-cancer to 0.0030 against 0.0026, either to 0.0051 against 0.0047, and bronchitis to 0.8624 against 0.8628 — every gap under half a percentage point, on a graph where the algorithm carries no proof that it should converge to anything at all. This is worth pausing on precisely because it is not the book's usual finding. Lauritzen and Spiegelhalter built the junction tree to buy an exactness guarantee on a network the size of a doctor's office chest-clinic example — eight variables, 256 configurations, an arithmetic problem 1988 hardware and 2026 hardware both finish before a human notices the delay. What their construction was actually paying for was never CPU time; it was correctness on graphs where cycles make brute-force loopy propagation unprincipled in a way neither paper cited here fully documents on its own: Kschischang and colleagues note only that cycles leave the sum-product algorithm with "no natural termination," not that loopy propagation actively oscillates, and Lauritzen and Spiegelhalter's 1988 construction predates loopy belief propagation entirely. It is the belief-propagation literature of the following two decades — on tree-reweighted and generalized message passing, working on graphs larger and more adversarial than this eight-node chest clinic — that established what those two papers alone do not: that ordinary loopy sum-product carries, in general, "no guarantees on the quality of the marginal beliefs, nor on the convergence of message passing." That distinction is the mathematics, not the hardware, and it is the reason the field spent the 1990s on junction trees and variational bounds rather than shrugging and running sum-product on the raw graph: a faster machine gets you the same wrong number faster. What the rerun shows is the empirical fact usually cited in that argument's favor — that on graphs with a single short cycle and reasonably strong local evidence, loopy belief propagation tends in practice to settle quickly and close to the exact marginal, which is exactly why, a decade after this paper, the field's engineers began using loopy propagation on turbo codes and Markov random fields anyway, guarantee or no guarantee.
By the middle 1970s, statisticians working in genetics, psychometrics, and survey sampling kept arriving, independently, at the same shape of problem. The data actually recorded were not the data the model was written for. A questionnaire had missing cells; a genetic cross could not distinguish two genotypes that produced the same phenotype; a mixture of two populations was sampled without anyone recording which population each individual came from. In every case a perfectly ordinary maximum-likelihood problem — write down a parametric density, differentiate its log with respect to the parameters, set the result to zero — became intractable not because the underlying model was exotic but because the observed data were a coarsened, marginalized, or censored version of some better-behaved "complete" data the model was really about. Arthur Dempster, Nan Laird, and Donald Rubin's 1977 paper to the Royal Statistical Society gave this family of problems a common name and a common algorithm. Their opening move was to notice that the difficulty was always the same difficulty: an easy likelihood existed, but it was a likelihood for data one did not have.
Dempster, Laird and Rubin's framework is deliberately abstract, and the abstraction is the point: it lets a missing questionnaire entry, a censored survival time, and an unobserved mixture-component label all be described by the same two objects. The paper writes that the term "incomplete data" implies "the existence of two sample spaces" — call them X and Y — "and a many-one mapping from" X "to" Y. The observed data y are a realization from Y; the corresponding "complete" data x lives in X but is not observed directly, only through the equation y = y(x), so that "x is known only to lie in" the subset of X "determined by the equation y = y(x), where y is the observed data." A family of complete-data densities f(x|φ) is postulated, and a corresponding "family of sampling densities g(y|φ)" is derived from it by the relation g(y|φ) = ∫ f(x|φ)dx, the integral taken over the set of complete data consistent with y. This integral is the whole difficulty. f(x|φ), the density of the complete data, is typically an easy exponential family whose maximum-likelihood estimate is a textbook formula. g(y|φ), the density actually available for inference, is that easy density integrated over a set that can be enormous: every way of completing the missing values, every assignment of data points to mixture components, every internal state sequence consistent with an observed output string. The likelihood one can write down cleanly is not the likelihood one can observe; the likelihood one can observe is an integral over the likelihood one cannot afford to compute pointwise.
Dempster, Laird and Rubin ground the abstraction in an example that predates their paper by more than a decade: a genetic-linkage problem due to C. R. Rao. "Rao (1965, pp. 368-369) presents data in which 197 animals are distributed multinomially into four categories, so that the observed data consist of" counts y1, y2, y3, y4 summing to 197. "A genetic model for the population specifies cell probabilities" of the form (½ + ¼π, ¼(1−π), ¼(1−π), ¼π) for a single unknown linkage parameter π between 0 and 1. Maximizing the multinomial log-likelihood g(y|π) directly requires differentiating a sum of four terms in which π enters awkwardly through a compound cell probability ½ + ¼π — doable by Fisher scoring, as Rao himself had done, but doable only by an iterative Newton-type calculation that gives no particular insight into why the algorithm converges or how fast. The EM move is to imagine that y1 is itself a coarsening of two unobserved counts, x1 and x2, that would appear in a hypothetical five-category multinomial with cell probabilities (½, ¼π, ¼(1−π), ¼(1−π), ¼π) — splitting the awkward compound cell into two clean ones so that the complete-data likelihood is an easy exponential family in π. Because x1 and x2 are not observed, only their sum y1 = x1 + x2 is, the observed-data likelihood is recovered by summing the complete-data likelihood over the 126 pairs (x1, x2) with x1 + x2 = 125 (the paper works with 125 as the observed value of y1, the remaining categories y2, y3, y4 fixed at 18, 20 and 34) — precisely the integral of equation (1.1) collapsed to a finite sum, but a sum whose 126 terms would have to be evaluated and added for every trial value of π if one insisted on maximizing g(y|π) directly. The point of the worked example is that one never has to: the algorithm alternates estimating the conditional expectation of x1 given the current guess of π, then re-estimating π by maximum likelihood as though that expectation were the observed complete data — and, started from π = 0.5, it reaches four-decimal-place agreement with the closed-form quadratic solution for the maximum-likelihood estimate — π* ≈ 0.6268214980 — by the third or fourth iteration, and agreement to seven decimal places by the eighth.
The Rao example is small enough — 126 terms in a sum over integer pairs — that direct evaluation was never really the obstacle; Rao's own Fisher-scoring solution shows the likelihood could be maximized without any notion of complete data at all. The example is illustrative, not the reason EM matters. The reason is what happens when the same abstraction is applied to a finite mixture: N independent observations, each drawn from one of K component densities, with the component label unobserved. The complete data are the pairs (observation, label); the complete-data likelihood, conditional on knowing every label, is a trivial product of within-component likelihoods, each an easy exponential family. The observed-data likelihood sums out every label independently, so that the integral in equation (1.1) becomes a sum over K^N label assignments — a quantity no scoring algorithm evaluates term by term for any N of practical size. The same shape recurs, Dempster, Laird and Rubin note, in the concluding example of their own Section 4.3. There the labels are not drawn independently and identically as in the ordinary finite mixture; instead, DLR write, the indicators "are not independently and identically distributed, but rather are specified to follow a Markov chain," with the complete-data likelihood built from a matrix of transition probabilities and a vector of initial-state probabilities — a structure they credit to Baum and Eagon (1967), Baum et al. (1970) and Baum (1972). DLR describe this only in the language of transition matrices; they do not use the words "hidden Markov model" or "forward-backward algorithm," which belong to the later vocabulary that grew up around Baum's own papers. But the object is the same one: what would come to be called a hidden Markov chain, whose EM instantiation is the forward-backward algorithm, with the complete data a state sequence and the sum in (1.1) running over every path through the state space. The same combinatorial shape recurs again in Hartley's 1958 multinomial examples and Sundberg's 1974 exponential-family identities, all of which the 1977 paper folds into one E-step/M-step template. This is the distinction the rest of the chapter turns on. In the Rao example, the intractability was contingent: a slightly cleverer parameterization, or Rao's own scoring iteration, made it disappear. In the mixture and hidden-Markov cases the intractability is structural — the sum is over a space whose size grows exponentially with the sample, and no reparameterization removes it, because there is no way to name a small sufficient statistic for "which observation came from which component" without already knowing the labels. That distinction — an intractable sum that is a bookkeeping inconvenience versus one that is a combinatorial wall — is exactly the distinction this book asks of every computational limitation it encounters, and here the answer is unambiguous: an exact sum over exponentially many label assignments does not get easier on faster hardware. It bound Dempster, Laird and Rubin's readers in 1977 and it binds a 2026 machine just as absolutely; what EM offers, and what Section 9.2 derives, is not a way around the sum but a way to climb the likelihood surface without ever summing it in full.
Dempster, Laird and Rubin build the algorithm in three nested levels of generality, and it is worth climbing all three, because the mixture model the rest of this section derives lives on the middle level, not the simplest one. At the most restricted level, the complete-data density is a regular exponential family, f(x|φ) = b(x)exp(φt(x)ᵀ)/a(φ), with φ a natural parameter and t(x) a vector of complete-data sufficient statistics. Writing φ^(p) for the current fit, the cycle is stated as two steps: "E-step: Estimate the complete-data sufficient statistics t(x) by finding t(p) = E(t(x)|y, φ(p))" and then "M-step: Determine φ(p+1) as the solution of the equations E(t(x)|φ) = t(p)" — that is, re-estimate φ by ordinary maximum likelihood exactly as if t(p) had been observed directly. The Rao linkage example of Section 9.1 is an instance of this simplest level: the complete-data sufficient statistic is the split cell count x1, the E-step replaces it by its conditional expectation, and the M-step re-solves the one-parameter multinomial likelihood equation. Most models a working statistician actually wants — finite mixtures among them — are not this simple, because the mapping from the natural exponential-family parameter back to the parameters one cares about (mixing proportions, component means, component covariances) is curved rather than linear. Dempster, Laird and Rubin handle this by stripping out the exponential-family assumption entirely and defining the algorithm at its most general level directly in terms of a conditional expectation of the complete-data log-likelihood. They introduce "a new function Q(φ'|φ) = E(log f(x|φ')|y,φ)" and define the iteration φ(p) → φ(p+1) by "E-step: Compute Q(φ|φ(p)). M-step: Choose φ(p+1) to be a value of φ in Ω which maximizes Q(φ|φ(p))." The heuristic they give for why this is the right thing to maximize is direct: one would like to choose φ to maximize log f(x|φ), the complete-data log-likelihood, but x is not observed, so one maximizes instead its conditional expectation given the data actually in hand and the current parameter estimate.
The reason this indirect maximization is safe — the reason climbing Q instead of the true log-likelihood L(φ) = log g(y|φ) cannot make things worse — is a short argument from Jensen's inequality, and it is worth reproducing because Section 9.3's rerun leans on exactly this guarantee. Write k(x|y,φ) = f(x|φ)/g(y|φ) for the conditional density of the complete data given what was actually observed, so that L(φ) decomposes as L(φ) = log f(x|φ) − log k(x|y,φ) for any x. Taking conditional expectations of both sides given y and some fixed φ turns this into L(φ') = Q(φ'|φ) + H(φ'|φ), where H(φ'|φ) = E(log k(x|y,φ')|y,φ). Dempster, Laird and Rubin's Lemma 1 states that H(φ'|φ) ≤ H(φ|φ) for every pair (φ',φ), with equality only when the two conditional densities coincide almost everywhere — and the proof is one line: "Formula (3.3) is a well-known consequence of Jensen's inequality." Subtracting, L(φ(p+1)) − L(φ(p)) = [Q(φ(p+1)|φ(p)) − Q(φ(p)|φ(p))] + [H(φ(p)|φ(p)) − H(φ(p+1)|φ(p))]; the first bracket is non-negative because φ(p+1) was chosen to maximize Q(·|φ(p)), and the second bracket is non-negative by Lemma 1. Every EM step therefore increases the observed-data log-likelihood, or leaves it unchanged only at a stationary point — a guarantee that holds for the whole family the paper calls GEM algorithms, generalized EM, defined only by the weaker requirement that φ(p+1) improve Q rather than maximize it outright. Dempster, Laird and Rubin state the corollary plainly for the case that matters in practice: if φ(p) converges to some φ*, then φ* is a fixed point of the M-step, which combined with the exponential-family E-step forces the derivative of the log-likelihood to vanish there. None of this says the fixed point is a global maximum — mixture likelihoods are riddled with local ones, a fact Section 9.3's rerun will make concrete — only that the climb never goes downhill.
Dempster, Laird and Rubin list finite mixtures among their examples, but they do not carry the normal-mixture algebra through in the paper itself; the summary promises "applications to grouped, censored or truncated data, finite mixture models, variance component estimation, hyperparameter estimation, iteratively reweighted least squares and factor analysis," and when the text reaches mixtures it defers to work already in print — "Day (1969) considered mixtures of two multivariate normal populations with a common unknown covariance matrix, while Wolfe (1970) studied mixtures of binomials and mixtures of arbitrary multivariate normal distributions," with Hasselblad's 1966 and 1969 papers credited as the earliest instances of the same iteration. What follows is the derivation those papers worked out and DLR's own apparatus licenses, written in the Q(φ'|φ) notation of the previous block rather than in Day's or Wolfe's; where DLR is silent on the Gaussian-mixture algebra specifically, this is the reader carrying their Section 2 template the rest of the way, not a quotation from the paper. Suppose y1,...,yN are drawn independently, each from one of K multivariate normal populations, and the label of which population produced yᵢ is not recorded. The complete data for observation i is the pair (yᵢ, zᵢ), where zᵢ = (zᵢ1,...,zᵢK) is a one-of-K indicator vector, zᵢk = 1 if yᵢ came from component k and 0 otherwise — precisely DLR's abstract distinction between the observed y and the complete x, with x = {(yᵢ,zᵢ)} and the many-one map x → y simply forgetting the labels. If the labels were known, the complete-data log-likelihood would separate into a multinomial piece and K independent normal pieces: log f(x|φ) = Σᵢ Σₖ zᵢk [log πₖ + log φ_N(yᵢ; μₖ, Σₖ)], where φ_N(·;μₖ,Σₖ) is the multivariate normal density and φ = (π₁,...,π_K, μ₁,...,μ_K, Σ₁,...,Σ_K) is the full parameter, subject to Σₖπₖ = 1. This is an exponential family in the zᵢk and yᵢzᵢk and yᵢyᵢᵀzᵢk jointly, but the map from its natural parameters back to (π,μ,Σ) is curved, not linear — the case DLR's Section 2 flags as needing the general Q-function definition rather than the simple sufficient-statistic E-step of the regular exponential family. The E-step computes Q(φ'|φ) = E(log f(x|φ')|y,φ), and because log f(x|φ') is linear in the unobserved zᵢk, this reduces to replacing each zᵢk by its conditional expectation given the data and the current fit. Since zᵢk is a 0/1 indicator, that conditional expectation is a posterior probability, obtained by Bayes's rule from the current component densities and mixing weights: γᵢk := E(zᵢk | yᵢ, φ) = πₖ φ_N(yᵢ;μₖ,Σₖ) / Σⱼ πⱼ φ_N(yᵢ;μⱼ,Σⱼ). This is the entire E-step: N·K numbers, one responsibility per observation per component, computed once each iteration — in place of DLR's equation (1.1), which would require summing the complete-data likelihood over every one of the K^N ways of assigning the N observations to K labels. The M-step maximizes Q(φ|φ(p)) = Σᵢ Σₖ γᵢk(p) [log πₖ + log φ_N(yᵢ;μₖ,Σₖ)] over φ subject to Σπₖ=1. Introducing a Lagrange multiplier for the constraint and differentiating with respect to πₖ, μₖ and Σₖ in turn — each term in the sum over k separates cleanly because the constraint is the only thing coupling the components — gives the closed-form updates πₖ(p+1) = (1/N) Σᵢ γᵢk(p), μₖ(p+1) = Σᵢ γᵢk(p) yᵢ / Σᵢ γᵢk(p), Σₖ(p+1) = Σᵢ γᵢk(p) (yᵢ−μₖ(p+1))(yᵢ−μₖ(p+1))ᵀ / Σᵢ γᵢk(p), which are exactly weighted maximum-likelihood estimates of a mixing proportion, a mean and a covariance, with each observation's weight in component k equal to its posterior responsibility for that component. Because this M-step solves its first-order conditions in closed form, DLR's exponential-family E-step/M-step template applies cycle by cycle even though the overall curved family required the general Q-function definition to set the E-step up in the first place — the curvature lives in the coupling across components, not within any one component's regular-exponential-family update.
Rubin and Thayer's 1982 paper is worth reading alongside the mixture derivation because it applies the identical DLR apparatus to a different choice of complete data and shows the machinery running on real numbers rather than abstract symbols. Maximum-likelihood factor analysis, they observe at the outset, "can be conceptualized as maximum likelihood estimation in a multivariate normal model with missing data," citing "Dempster, Laird, & Rubin (1977) section 4.7" directly — the missing data here are not component labels but the factor scores themselves, the unobserved q-dimensional vector Zᵢ underlying each observed p-dimensional row Yᵢ. "The EM algorithm treats the factor matrix Z as missing data" and iteratively re-maximizes the likelihood as though Z had been observed, which is precisely the DLR template: an easy complete-data likelihood in (Y,Z) jointly normal, an E-step that computes the conditional expectation of the complete-data sufficient statistics — here the matrices Cyy, Cyz and Czz — given the observed Y and the current loadings, and an M-step that re-estimates the loading matrix β and uniquenesses τ² by ordinary regression formulas as though those expected statistics were the observed ones. Applied to Jöreskog's nine-variable, four-factor example, run from three different starting solutions for fifty iterations, the paper reports that "the estimates for z 2 found by EM agree with those found by J6reskog's program from the same starting values" — a numerical check on the monotonicity theorem of the previous block — while also noting the flip side of a likelihood surface with multiple maxima: different starting solutions can converge to different fixed points, since nothing in the monotonicity proof rules out more than one stationary point. The paper's own framing of what EM buys, independent of any particular application, doubles as the cleanest statement of why Section 9.3 reruns a mixture rather than a factor analysis. Citing "the general theory of EM algorithms given in Dempster, Laird, and Rubin [1977]," it states that this theory "proves not only that each iteration of EM increases the likelihood, even if starting from a point where the likelihood is not convex, but also that if an instance of the algorithm converges, it converges to a (local) maximum" — a guarantee about the shape of the climb, not about which peak it reaches. Mixtures make the multiple-peaks problem far more visible than factor analysis does, because permuting the K component labels always produces K! numerically distinct fixed points of identical likelihood, and nothing in DLR's convergence theory — nor in Rubin and Thayer's application of it — distinguishes among them, or between them and a poorer local maximum reached from a different start. That is not a defect a faster machine repairs; it is a property of the surface EM is climbing, the same distinction Section 9.1 drew between an intractable sum and a merely inconvenient one, applied now to the likelihood's shape instead of its computability.
Section 9.2 derived two things that can be checked independently, and this section checks both. The first is a claim about a number: that Dempster, Laird and Rubin's genetic-linkage example, cycled through their equations (1.4) and (1.5) from a starting value of π = 0.5, reproduces their own Table 1 to the precision they printed it, and that the ratio of successive deviations from the maximum-likelihood estimate settles to the constant they report. This is the cheapest possible rerun — 197 animals, four observed counts, one scalar parameter — and it is worth doing exactly because it is cheap: if a hand-checked 1977 table cannot be reproduced from the paper's own stated equations on any machine built since, something is wrong with the reading of the paper, not with 1977 hardware. The second claim is qualitative rather than numerical, and it is the one the previous section flagged but did not demonstrate: that the monotone-likelihood guarantee of Lemma 1 holds exactly as advertised, while the shape of the mixture likelihood surface it climbs is riddled with distinct fixed points of unequal height, so that which one an EM run reaches depends on where it started and nothing else. Rubin and Thayer's 1982 factor-analysis rerun of the DLR machinery reported this qualitatively — running EM from three different starting values and finding, in their words, that "three different starting values lead to three essentially different (i.e., different z 2) solutions" — z² being their symbol for the uniqueness matrix, i.e. this is a claim about non-identifiable parameter estimates, not about differing fit statistics — a fact they called "quite interesting" because it "complicates the interpretation of any solution," without quantifying how often, or how far apart the fixed points are, on any concrete example. This section builds a small synthetic Gaussian mixture using exactly the E-step and M-step closed forms derived in Section 9.2 — γᵢk, and the weighted mean/covariance/mixing-proportion updates — verifies the monotonicity claim iteration by iteration, and then runs the same algorithm from twenty different random starts to see how many distinct answers a deterministic climb on a fixed dataset actually produces.
Rao's data (Section 9.1) are four multinomial counts, y = (125, 18, 20, 34) out of 197, with cell probabilities (½+π/4, (1−π)/4, (1−π)/4, π/4). Splitting the first cell into two hypothetical counts x₁ + x₂ = y₁ with probabilities ½ and π/4 gives the complete-data problem DLR actually iterate: an E-step estimating x₂⁽ᵖ⁾ = y₁·(π⁽ᵖ⁾/4)/(½+π⁽ᵖ⁾/4), and an M-step re-solving π⁽ᵖ⁺¹⁾ = (x₂⁽ᵖ⁾+y₄)/(x₂⁽ᵖ⁾+y₂+y₃+y₄) as an ordinary binomial maximum-likelihood equation. Coding exactly these two closed forms and cycling from π⁽⁰⁾=0.5 — the paper states "the algorithm moved for eight steps as displayed in Table 1" — reproduces, iteration by iteration: 0.500000000, 0.608247423, 0.624321050, 0.626488879, 0.626777322, 0.626815632, 0.626820719, 0.626821394, 0.626821484. Against the printed table — 0·500000000, 0·608247423, 0·624321051, 0·626488879, 0·626777323, 0·626815632, 0·626820719, 0·626821395 — the two sequences agree to eight or nine decimal places throughout, the residual last-digit disagreements consistent with 1977 hand or desk-calculator rounding rather than any divergence in the recursion itself. The closed-form maximum-likelihood estimate the paper derives algebraically from the fixed-point equation, π* = (15+√53809)/394 ≈ 0.6268214980, matches a direct evaluation in floating point, 0.6268214978709824, and the EM iterate after eight steps, 0.626821484, sits within 2×10⁻⁸ of it — matching to seven decimal places by the eighth iteration, exactly the rate of agreement Section 9.1 described. The ratio of successive deviations π⁽ᵖ⁾−π* is not constant from the first step — it runs 0.146, 0.135, 0.133 — but by the third iteration it has settled, exactly as claimed: "The ratios are essentially constant for p ~ 3," and a rerun computing that ratio directly gets 0.133024, 0.132811, 0.132783, 0.132779, 0.132779, 0.132779 for p = 3 through 8, matching the paper's reported asymptotic rate of 0.132778 (computed independently, in Section 3 of the paper, from the ratio of missing to observed information at π*) to five decimal places.
The linkage example is a single scalar climbing along a one-dimensional curve; it cannot show what Section 9.2 flagged as the interesting failure mode, because a one-parameter problem with a fixed-point equation this simple has nowhere else to converge to. The second rerun uses the K = 3 Gaussian-mixture E-step and M-step derived in Section 9.2 — γᵢk = πₖφ_N(yᵢ;μₖ,Σₖ)/Σⱼπⱼφ_N(yᵢ;μⱼ,Σⱼ) in the E-step, and the weighted closed-form updates for πₖ, μₖ, Σₖ in the M-step — coded directly from those formulas rather than from a library implementation, on 300 points drawn from three well-separated bivariate normal components (means at (0,0), (5,5), and (0,6), each with covariance 0.8·I, mixing weights 0.3/0.4/0.3). Running the recursion from a single random initialization of the three means (points chosen at random from the data, unit covariances, uniform mixing weights) and logging the observed-data log-likelihood L = Σᵢ log[Σₖπₖφ_N(yᵢ;μₖ,Σₖ)] every iteration gives, for the first six steps, −2832.8551, −1258.0650, −1223.1706, −1216.0614, −1214.9169, −1213.2269, climbing to a converged value of −1207.0096 after fifty iterations. Checking every successive pair of iterations against the Jensen's-inequality guarantee proved in Section 9.2 — that L(φ⁽ᵖ⁺¹⁾) − L(φ⁽ᵖ⁾) can never be negative — an explicit comparison over the full run finds every increment non-negative to within 10⁻⁸, the residual only floating-point noise in the log-sum-exp evaluation. This is Dempster, Laird and Rubin's Lemma 1 doing exactly what it is licensed to do: the climb never goes downhill, on this data, on this run, to eight decimal places.
What the single run does not show is what happens across runs, and this is where Rubin and Thayer's warning turns from a sentence into a number. Repeating the identical algorithm twenty times on the identical 300 points, varying only which three data points seed the initial means, produces converged log-likelihoods of (sorted): −1190.47, −1188.94, −1186.54, −1173.95, −1172.90, −1172.13, −1169.74, −1168.50, and then twelve runs — twelve out of twenty — landing on −1073.46. The twelve-way tie is not a coincidence of rounding: inspecting the fitted means from four of those twelve runs directly gives, in every case, components at (−0.02,−0.04), (0.03,6.04), and (4.91,5.01) — the same triple, up to which row of the output happens to list which component, and matching the three generating means (0,0), (0,6), (5,5) to within the sampling noise of 300 draws. That agreement up to row order is DLR's own point about label permutation made concrete: with K = 3 components there are 3! = 6 numerically distinct parameter vectors that describe the identical mixture distribution and therefore the identical likelihood, so the "twelve runs converge to one answer" is really some smaller number of genuinely distinct optima, each reached from multiple different random seeds, redescribed under different component orderings. The other eight runs are not relabelings of the good solution; they are worse solutions outright, each a legitimate fixed point of the M-step — the derivative of the log-likelihood vanishes there exactly as Section 9.2's corollary requires — reached from a starting configuration in which two of the three initial means happened to fall inside the same true cluster, so that the algorithm spent its whole run explaining two real components with one fitted component and splitting the third real component in two. No amount of additional iteration escapes this: each of the eight runs had converged, by the paper's own stopping criterion (successive log-likelihoods agreeing to 10⁻⁸), tens of iterations before the twenty-run sweep ended. This is the distinction Sections 9.1 and 9.2 both drew and this rerun now makes numeric: an intractable sum over label assignments is a wall no faster machine climbs over, but a monotone hill-climb converging to the wrong hill is a different kind of limitation entirely, one that a 2026 datacentre inherits unchanged from 1977, because it is a fact about the likelihood surface of a Gaussian mixture, not about the arithmetic used to climb it. What 2026 hardware buys is the ability to run the twenty restarts in well under a second rather than choose one and hope — multiple restarts, not a faster restart, is the fix DLR's own convergence theory implies and Rubin and Thayer's isolated anecdote about "different starting solutions" only gestured at.
A decision tree is the least exotic model this book will derive: a nested sequence of if-then tests ending in a class label, learned by repeatedly splitting a training set into purer and purer subsets. It is also, on the historical record, the most independently reinvented one. Statisticians working on survey data, pattern-recognition engineers reading satellite imagery, and machine-learning researchers trying to automate expert knowledge all arrived at the same recursive-partitioning idea within a few years of one another, largely without citing each other. As Sreerama Murthy's 1998 survey of the field observes, "Work on decision tree induction in statistics began due to the need for exploring survey data (Fielding, 1977). Statistical programs such as AID (Sonquist, Baker and Morgan, 1971), MAID (Gillo, 1972), THAID (Morgan and Messenger, 1973) and CHAID (Kass, 1980) built binary segmentation trees aimed towards unearthing the interactions between predictor and dependent variables." A separate line of work grew out of remote sensing: "Pattern recognition work on decision trees was motivated by the need to interpret images from remote sensing satellites such as LANDSAT in the 1970s (Swain and Hauska, 1977)." And a third line grew out of the expert-systems bottleneck this book's first volume already tells the story of: "Decision trees in particular, and induction methods in general, arose in machine learning to avoid the knowledge acquisition bottleneck (Feigenbaum, 1981) for expert systems." By the early 1980s these threads converged on two canonical systems that this chapter treats as the twin reference points for everything that follows: Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone's CART, and J. Ross Quinlan's ID3. A 1993 paper on oblique trees summarizes the convergence bluntly: "In the 1980's, Breiman et al.'s book on classification and regression trees (CART) and Quinlan's work on ID3 (Quinlan, 1983, 1986) provided the foundations for what has become a large body of research on one of the central techniques of experimental machine learning."
Despite their different lineages, CART and ID3 (and the dozens of descendants each spawned) share a single algorithmic skeleton, which the literature calls top-down induction of decision trees, TDIDT. Murthy, Kasif, and Salzberg's paper on oblique trees and the OC1 algorithm — filed in this corpus under the misleading label "Heath 1993," though its actual authors are the OC1 team, not Heath, whose separate SADT system it cites only as related work — states it as cleanly as any source in the corpus: "Algorithms for inducing decision trees follow an approach described by Quinlan as top-down induction of decision trees (1986). This can also be called a greedy divide-and-conquer method." The outline itself has four steps: "1. Begin with a set of examples called the training set, T. If all examples in T belong to one class, then halt. 2. Consider all tests that divide T into two or more subsets. Score each test according to how well it splits up the examples. 3. Choose (“greedily”) the test that scores the highest. 4. Divide the examples into subsets and run this procedure recursively on each subset." Every design decision that distinguishes one tree-growing algorithm from another lives inside this skeleton rather than beside it. Step 2 asks: what is the space of candidate tests, and by what score is a test judged good? Step 3 asks: having scored all candidates, do we take the single best, or hold several open? Step 4 asks: when do we stop recursing — at purity, at a fixed depth, at a minimum node count — and do we prune afterward? CART and ID3 answer the first question differently (axis-parallel thresholds on ordered attributes versus axis-parallel splits on discrete or continuous attributes scored by different impurity measures), answer the second identically (always greedy, never backtracking), and answer the third with different stopping rules that this chapter's second section takes up directly. The rest of this section is about step 2 and step 3: what a splitting criterion is, why the major candidates turn out to behave alike, and why "greedily" in step 3 is not a concession to limited compute but the only tractable move available at all.
Step 2 of the TDIDT skeleton — scoring candidate splits — is where the historically separate lineages actually differ, at least on paper. Murthy's survey groups the splitting rules into families. One family comes straight from information theory: "Examples of this variety are rules based on Shannon's entropy," with Quinlan's ID3 measuring "the reduction in entropy due to splitting each individual node" — information gain, in the vocabulary this book has already used for coding-theoretic arguments in earlier chapters. A second family comes from statistics by way of population genetics: CART's default criterion is "the Gini index of diversity," a measure of how mixed the classes are within a node, borrowed from a measure of diversity used elsewhere in the biological and social sciences. Breiman and his coauthors already knew this measure had a weakness: "Breiman et al. pointed out that the Gini index has difficulty when there are a relatively large number of classes," which is why CART also offers the twoing rule as an alternative. What makes this taxonomy less consequential than it looks is a finding the survey states without much ceremony: "It is not surprising that most existing splitting rules are functionally equivalent." Entropy-based gain, the Gini index, and their many descendants — gain ratio, distance measures, statistical significance tests — tend to rank candidate splits in nearly the same order on real data, because they are all, at bottom, concave measures of node impurity being compared before and after a proposed split. The historical disputes between the statistics camp and the machine-learning camp over which criterion is "correct" turn out to be disputes over which of several closely related yardsticks to use, not over whether the yardstick approach itself is sound. The genuinely hard problem — addressed below — is not which impurity measure to pick, but what greedy, one-split-at-a-time optimization against any of them can and cannot guarantee.
Step 3 of the TDIDT skeleton asks the sharpest question of the four: having scored every candidate split, does the algorithm commit to the single highest-scoring test, or does it hold several candidates open — via lookahead, backtracking, or exhaustive search over subtrees — before committing? Every major tree-growing system descended from CART and ID3 answers this the same way: greedily, one split at a time, never revisited. The natural objection is that this is a shortcut forced by the hardware of the 1970s and 1980s, and that a machine with billions of times the throughput should simply search deeper. The record — both empirical and theoretical — says the objection fails on both counts. Empirically, lookahead has been tried repeatedly, across every lineage this section has traced. Murthy's survey catalogues attempts spanning statistics, pattern recognition, tree-structured vector quantization, Bayesian trees, and neural-network trees, in addition to machine learning proper, and reports a uniform verdict: "Most of these studies indicate that lookahead does not cause considerable" gains "over greedy induction." Murthy and Salzberg's own controlled study of one-level lookahead went further than finding no benefit — they found active harm: "Murthy and Salzberg (1995b) demonstrate that one-level lookahead does not help build" significantly "better trees and can actually worsen the quality of trees, causing pathology." Deeper search over the same greedy scoring function does not reliably buy a better tree; sometimes it buys a worse one. Theoretically, the ceiling is sharper still, and it is mathematics rather than machinery. Hyafil and Rivest showed in 1976 that finding the single best tree is not merely expensive but intractable in the formal sense: the survey states that they "proved that the problem of building optimal decision trees from decision tables" is optimal "in the sense of minimizing the expected number of tests required to classify an unknown sample, is NP-Complete." Later results extended the same conclusion to cost-sensitive trees, storage-optimal trees, and trees judged by several other measures, none of them tractable exactly. This is the distinction the rest of this book returns to again and again: a slow run on 1970s hardware is a fact about the machine, but an NP-completeness proof is a fact about the problem, and it binds a 2026 datacenter exactly as it bound the workstations of 1976. No increase in clock speed turns exhaustive search over decision trees into a polynomial-time algorithm. "Greedily" in step 3 is therefore not the corner CART and ID3 cut because they were slow; it is the corner every tree-growing algorithm before or since has had to cut, because the alternative is provably worse than intractable on its own terms as often as it is better. What step 3 leaves open is when to stop applying it — when a node is pure enough, small enough, or unpromising enough that further greedy splitting should halt, and whether the resulting tree should then be cut back. That is the subject of this chapter's second section.
Section 2.1's TDIDT skeleton left step 2 — scoring candidate tests — underspecified in one respect: what counts as a candidate test in the first place. For ID3 and CART in their original, axis-parallel form, a test at a node checks a single attribute against a threshold, xi > k, and the space of candidates is small enough to search exhaustively: for n training instances and d attributes there are at most n·d distinct axis-parallel splits, one for each attribute at each observed value. Quinlan's ID3 and Breiman, Friedman, Olshen and Stone's CART both exploit this directly — "axis-parallel methods such as C4.5 (Quinlan, 1993a) and CART (Breiman et al., 1984) can exhaustively search for the best split at each node" — which is why the choice of impurity measure discussed in this chapter's first section is the only real design freedom in the axis-parallel case. Murthy, Kasif and Salzberg's 1994 paper on the OC1 system — filed in the corpus under the misleading label "Heath 1993," though its actual authors are the OC1 team — generalizes the test itself. Instead of a single attribute compared to a threshold, a node may test a linear combination of all the attributes at once: "let an example take the form X = x1, x2, . . . , xd, Cj where Cj is a class label and the xi's are real-valued attributes. The test at each node will then have the form: d∑i=1 aixi + ad+1 > 0 (1) where a1, . . . , ad+1 are real-valued coefficients." Because such a test is geometrically a hyperplane at an oblique angle to the coordinate axes, rather than one perpendicular to a single axis, the paper calls the resulting model an oblique decision tree, and notes the axis-parallel case is the special case ai = 0 for all but one coefficient: "It is clear that these are simply a more general form of axis-parallel trees, since by setting ai = 0 for all coefficients but one, the test in Eq. 1 becomes the familiar univariate test." Where axis-parallel trees partition the attribute space into hyper-rectangles, oblique trees partition it into arbitrary polyhedra — potentially far fewer of them for a concept that is itself defined by a handful of slanted decision surfaces. Generalizing the test destroys the one property that made step 2 easy: exhaustive search. Where axis-parallel splits number n·d, the oblique splits number at most 2^d·C(n,d) — 2 to the d, times "n choose d" — the paper's own count of "all 2d· (n d ) distinct hyperplanes" that a brute-force method would have to enumerate and choose the best from, "as is done in axis-parallel decision trees." The C(n,d) term counts the ways to pick a subset of d points to define a candidate hyperplane; the 2^d term counts the ways each such hyperplane can be perturbed to separate that subset in all possible ways. Enumerating and scoring all of them is no longer a matter of a linear pass over attribute values; it is exponential in the number of attributes d and combinatorial in the number of points n. And the paper is explicit that this is not merely inconvenient but provably intractable: "Heath (1992) proved that the following problem is NP-hard: given a set of labelled examples, find the hyperplane that minimizes the number of misclassified examples both above and below the hyperplane." Its consequence is stated directly: any method for "finding the optimal oblique split is likely to have exponential cost (assuming P̸ = NP)." A second, independent hardness result closes off the alternative of searching for the best small tree directly rather than the best single split: "This conclusion follows from the work of Blum and Rivest (1988)... in d dimensions (i.e., with d attributes) the problem of producing a 3-node oblique decision tree that is consistent with the training set is NP-complete." Both results are about the mathematics of the search space, not about the machines available to search it — a 2026 datacenter inherits them exactly as the 1993 workstation did. What follows in this section is the sequence of heuristics — CART's linear-combination coordinate descent, and OC1's randomized generalization of it — built to make step 2 tractable in practice once exhaustive search is ruled out in principle.
Section 2.2 built to two hardness results — Heath's proof that finding the optimal single oblique hyperplane is NP-hard, and Blum and Rivest's proof that even a 3-node oblique tree consistent with the data is NP-complete to construct — and then to the practical answer the field settled on: give up on optimality and search a single hyperplane's coefficients by coordinate descent (CART-LC) or by randomized restarts and perturbations (OC1). Murthy, Kasif and Salzberg's own Table 1 measured what that practical answer bought, on six real-world domains, using ten repetitions of five-fold cross-validation, reporting the mean and standard deviation of both classification accuracy and tree size in leaves. Table 2 measured a second, more elementary claim on artificial domains built around simple geometric concepts: that a single test of the form Σaᵢxᵢ + a_{d+1} > 0 can capture a concept that an axis-parallel tree can only approximate with a staircase of many small rectangular splits. This section reruns both kinds of experiment on modern hardware, using a modern axis-parallel baseline (scikit-learn's CART implementation) against a from-scratch oblique tree built to the same specification as Murthy et al.'s Eq. 1 test, but fit at each node by logistic regression rather than OC1's randomized coefficient search.
The one-dataset rerun uses the Wisconsin breast cancer diagnosis task, chosen deliberately because it is the same clinical problem behind one of Murthy et al.'s six domains — patients characterized by numeric attributes derived from a biopsy, labelled benign or malignant. It is not the same data file: their table used "a set of patients with breast cancer" compiled by Mangasarian and Bennett, nine numeric attributes and 683 entries; scikit-learn's bundled load_breast_cancer is the later, 1993 Wisconsin Diagnostic set of Street, Wolberg and Mangasarian, 569 instances described by thirty numeric attributes computed from digitized cell-nucleus images. The substitution swaps one Wolberg-lineage breast-cancer dataset for another rather than reaching for an unrelated modern corpus, but it is a substitution, and the numbers below should be read as a rerun of the experimental design, not a literal reproduction of Table 1's row. Features were standardized (zero mean, unit variance) before fitting, since the logistic-regression coefficient search used here — unlike CART-LC's or OC1's — has no built-in normalization step and diverges on the raw attribute scales. The methodology otherwise follows Murthy et al.'s Section 4.1 as closely as a from-scratch implementation allows: ten repetitions of five-fold stratified cross-validation, reporting the mean and standard deviation of accuracy and of tree size in leaves.
Rerunning both models on standardized features (569 instances, 30 attributes, ten repetitions of five-fold stratified cross-validation) gave: Axis-parallel (scikit-learn's CART, grown to purity, Gini impurity): accuracy 0.9258 ± 0.0218, tree size 19.2 ± 2.1 leaves. Oblique (single logistic-regression hyperplane per node, Eq. 1 form, grown to depth 6 or purity): accuracy 0.9801 ± 0.0116, tree size 2.0 ± 0.0 leaves. The qualitative finding matches Murthy et al.'s own: an oblique split that is allowed to weigh all thirty attributes at once needs far fewer of them than a test restricted to one attribute at a time, because malignancy in this data is better separated by a linear combination of the cell-nucleus measurements than by any single one thresholded on its own. The margin here is in fact wider than the paper's own tables typically showed for its six domains, and the reason points at what changed rather than at what stayed the same. OC1's per-node search was a randomized local search over hyperplane coefficients, built in 1993 to be affordable on the hardware of the day; the rerun instead fits each node's hyperplane by a convex logistic-regression solver that finds the global optimum of its own (differently regularized) objective in a fraction of a second. On this dataset a single well-fit hyperplane is enough to resolve the great majority of cases, which is why the oblique tree above averages almost exactly two leaves — one split, not the deeper multi-node oblique trees OC1 needed when its local search settled for less. The comparison is therefore not simply oblique-versus-axis-parallel; it is also a demonstration of how much of OC1's design was scaffolding around a per-node optimization problem that a 2026 convex solver dispatches essentially exactly.
The second rerun isolates the geometric point Table 2's artificial domains were built to make, stripped to its simplest case: a concept whose true boundary is a single hyperplane at an oblique angle to the coordinate axes. Two thousand points were drawn uniformly from [-2, 2]², labelled by the diagonal rule x + y > 0 — a boundary at exactly 45 degrees to both axes, so that no single-attribute threshold can align with it at all. An axis-parallel tree (scikit-learn's CART, grown without a depth limit, as ID3 and CART both do by default before pruning) needed 42 leaves to fit this training set to perfect accuracy, approximating the diagonal with a staircase of ever-smaller rectangles — visibly the "hyper-rectangle" partition Section 2.2 described, chasing a boundary the representation cannot express exactly no matter how many times it recurses. A single test of the oblique form a₁x₁ + a₂x₂ + a₃ > 0, fit once by logistic regression, reproduced the boundary in one split at 99.8 percent training accuracy and two leaves. This is the sharpest form of the claim in Section 2.2: the gap between axis-parallel and oblique trees on a diagonal concept is not a matter of search effort at all — no amount of additional axis-parallel splitting closes it, because the representation itself cannot express a slanted boundary except by approximation, while the oblique representation expresses it exactly with one test. More data or more compute make the axis-parallel staircase finer, never flat.
Two qualifications keep this rerun honest about what it does and does not show. First, the oblique tree implemented here is a simplification of OC1, not a reproduction of it: it fits each node's hyperplane once, by a convex solver, rather than by OC1's twenty randomized restarts and five random jumps per local minimum, a design built specifically to escape the local minima that a single deterministic search (like CART-LC's coordinate descent) can get stuck in. On these two problems a single global optimum sufficed, so the randomization bought nothing extra to rerun; on a harder domain, where the impurity surface is more jagged, OC1's own Section 4.3 experiments found that randomization mattered, and a from-scratch single-shot solver would not necessarily recover what multiple restarts do. Second, and more important for this book's argument: nothing in either rerun is a story about waiting. The paper itself is explicit that OC1's own heuristic search was never the slow part it needed a faster machine for — the authors state their implementation's complexity directly: "the current implementation of OC1 guarantees a worst-case running time that is only O(log n) times greater than the worst-case time for inducing axis-parallel trees (i.e., O(dn²log n) vs. O(dn²))." That guarantee held on the workstations of 1993 exactly as it holds on a 2026 datacenter, because it is a statement about an algorithm's polynomial complexity, not about a clock speed. What no machine of any era escapes is the object the heuristic was built to avoid searching exhaustively — Heath's proof that the single optimal hyperplane is NP-hard to find, and Blum and Rivest's proof that even a 3-node oblique tree consistent with the training set is NP-complete to construct. A 2026 datacenter reruns OC1's coordinate descent or randomized search in milliseconds instead of seconds; it does not turn the exhaustive search over all 2^d·C(n,d) oblique hyperplanes, or over all decision trees consistent with a training set, into a polynomial-time algorithm. The rerun's real finding is not that oblique trees are fast now — they always were, on hardware of any era, because both CART-LC and OC1 were built from the start to avoid the intractable search — but that a modern convex solver closes almost all of the remaining gap between OC1's randomized heuristic and the hyperplane it was heuristically approximating, on a task where that hyperplane happens to be close to linearly separable in the first place.
This chapter's derivations answer a question that was first posed as pure recursion theory and only later turned into an algorithm anyone would run. In Valiant's PAC framework — already this book's vocabulary since Chapter 6 — a concept class is strongly learnable if some polynomial-time algorithm drives the error arbitrarily close to zero with arbitrarily high confidence. Kearns and Valiant asked what happens if that demand is relaxed to something almost embarrassingly weak: an algorithm that need only "output an hypothesis that performs slightly better (by an inverse polynomial) than random guessing" on every distribution. They left open whether this weak learnability was in fact equivalent to the strong kind — the question Schapire's 1990 paper names outright: "This question was termed the hypothesis boosting problem since showing the notions are equivalent requires a method for boosting the low accuracy of a weak learning algorithm's hypotheses." The obvious first guess — just run the weak learner many times and let the outputs vote — had already been shown not to work: "Kearns (1988), considering the hypothesis boosting problem, gives a convincing argument discrediting the natural approach of trying to boost the accuracy of a weak learning algorithm by running the procedure many times and taking the 'majority vote' of the output hypotheses." A plain committee of independently-trained weak learners, in other words, is not enough; naive repetition produces more of the same weak hypothesis, not a strong one. Nor was the answer obvious on other grounds: Kearns and Valiant had already shown that under a fixed, restricted distribution, weak and strong learnability come apart. Schapire's result was the surprise the other way — for the unrestricted, distribution-free case the two notions coincide exactly, and the proof is constructive: "an explicit method is described for directly converting a weak learning algorithm into one that achieves arbitrary accuracy," by "filtering to modify the distribution of examples in such a way as to force the weak learning algorithm to focus on the harder-to-learn parts of the distribution." That single sentence is the seed of every boosting algorithm this chapter derives: a committee of weak models generalizes not because each member is independently trained on the same distribution — that route was already closed — but because each member is trained on a distribution reshaped to concentrate on the previous members' mistakes.
Schapire's original construction was correct but impractical — it needed advance knowledge of the weak learner's exact error rate and recursed in a way that blew up hypothesis size. Freund and Schapire's 1997 paper, which this chapter's next section will derive symbol by symbol as AdaBoost, replaces it with an adaptive scheme motivated by a story rather than a recursion. They open with a gambler who "decides he will wager a fixed sum of money in every race, but that he will apportion his money among his friends based on how well they are doing" — lacking clairvoyance about which friend will win most, he updates his confidence in each friend after every race rather than committing in advance. Translated into learning, the friends become not independently retrained copies of one weak learner but a genuine sequence: "On each round t=1, ..., T, the booster devises a distribution Dt over the set of examples, and requests (from an unspecified oracle) a weak hypothesis (or rule-of-thumb) ht with low error =t with respect to Dt." Two things distinguish this from Schapire's earlier construction and from a plain vote. First, the algorithm — called Hedge(β) in its general form — needs no prior knowledge of the weak learner's accuracy: "the new algorithm needs no prior knowledge of the accuracies of the weak hypotheses. Rather, it adapts to these accuracies and generates a weighted majority hypothesis in which the weight of each weak hypothesis is a function of its accuracy." Second, and more consequential for the question of why a committee generalizes, is what the reweighting buys: the paper proves that the error of the final combined hypothesis falls off exponentially in the number of rounds, provided each weak hypothesis clears even a small, possibly shrinking margin over chance guessing. The committee is not a device for averaging out noise across identical copies of a mediocre classifier, which Kearns had already shown gains nothing; it is a device for forcing a sequence of classifiers to specialize on whatever the sequence so far has gotten wrong, with a guarantee that the specializing pays off geometrically rather than merely additively.
Boosting's story is a PAC-theoretic one: turn a provably weak learner into a provably strong one by an adaptive sequence of reweighted training sets. Breiman's bagging, introduced the same year Freund and Schapire's paper appeared, answers a different question with a different tool, and the two setups are worth holding apart before Section 2 derives them side by side. Bagging asks not how to strengthen a weak learner but how to stabilize an already-decent one. Breiman's diagnosis names the relevant property directly: "Instability was studied in Breiman [1994] where it was pointed out that neural nets, classification and regression trees, and subset selection in linear regression were unstable, while k-nearest neighbor methods were stable," where instability means that "a small change in" the training set "can result in large changes in" the fitted predictor. Bootstrap aggregating — resampling the training set with replacement, refitting the base learner on each resample, and averaging or voting the results — is aimed squarely at that instability: "For unstable procedures bagging works well." The reason follows from an elementary application of Jensen's inequality to squared error, which Breiman spells out for numerical prediction: averaging over independently trained replicates never increases expected squared error, because expanding E(y − φ(x))² and applying E[Z²] ≥ (E[Z])² to the cross term shows the mean-squared error of the averaged predictor is always at least as good as the mean-squared error averaged over replicates. "The more highly variable" the base predictor is across replicate training sets, "the more improvement aggregation may produce. But" the aggregated predictor "always improves on" the single predictor's expected error — an unconditional guarantee that has no analogue anywhere in the boosting argument, which instead guarantees a rate of error decrease under a weak-learning assumption. For classification the guarantee is weaker and depends on a new notion, order-correctness — a base learner is order-correct at an input x if the class it predicts most often across replicate training sets is the class that is in fact most probable there. "If a predictor is good in the sense that it is order-correct for most inputs x, then aggregation can transform it into a nearly optimal predictor." But the guarantee cuts both ways: "unlike the numerical prediction situation, poor predictors can be transformed into worse ones. The same behavior regarding stability holds. Bagging unstable classifiers usually improves them. Bagging stable classifiers is not a good idea." A committee built this way generalizes for a reason that has nothing to do with weak learnability or adaptive reweighting: it exploits variance in the base learner across resamplings of a fixed training set, and where that variance is small or absent, bagging has nothing to offer.
Both stories — boosting's exponential error bound under weak learnability, bagging's Jensen's-inequality guarantee under instability — leave a puzzle neither one answers on its own, and it is this puzzle that Section 2 must resolve before the chapter can call either algorithm understood. Boosted committees keep improving on held-out data long after they have already driven training error to zero, which is exactly the regime in which every complexity-penalized generalization bound this book has derived since Chapter 6 predicts overfitting should begin. Schapire, Freund, Bartlett, and Lee's 1998 paper states the observation plainly: "both boosting and bagging eventually achieve" perfect or nearly perfect accuracy on the training sets, "but the generalization error for boosting is better. The explanation for this is evident from the margin distribution graphs where we see that, for boosting, far fewer training examples have margin close to zero" — that is, once every training point is already classified correctly, boosting continues driving the vote's margin of victory upward, and it is this margin, not the training error, that governs the generalization bound their paper derives. This account competes with an older one. "One of the main explanations for the improvements achieved by voting classifiers is based on separating the expected error of a classifier into a bias term and a variance term," where "the bias term measures the persistent error of the learning algorithm... that would remain even if we had an infinite number of independently trained classifiers" and "the variance term measures the error that is due to fluctuations that are a part of generating a single classifier" — a story that fits bagging (built explicitly to reduce variance across bootstrap replicates) far more naturally than boosting. The authors test the bias-variance account directly against boosting and find it insufficient: on synthetic data engineered to isolate the two terms, "boosting is doing more than reducing variance. For instance, on 'ringnorm,' boosting decreases the overall error of the stump algorithm from 40.6% to 12.2%, but actually increases the variance." Breiman's own random forests paper, read alongside this one, offers a third and complementary vocabulary for exactly the same question, applied now to a committee with no adaptive reweighting at all: it defines a margin function for an ensemble of randomized trees and derives a bound on the generalization error in terms of two measurable quantities, "the strength of the individual tree classifiers and a measure of the dependence between them" — an ensemble generalizes well to the extent its members are individually accurate and mutually decorrelated, a formulation general enough to cover bagged trees, boosted stumps, and randomized-feature forests within a single inequality. Section 2 takes up the margin, the bias-variance decomposition, and the strength-correlation bound in turn, and derives each in the notation its own paper uses.
Freund and Schapire derive AdaBoost as a special case of a more general on-line allocation algorithm, Hedge(β), and the derivation is worth following in that order because it explains where the exponential weight update comes from. Hedge(β) maintains a weight vector w^t = (w^t_1, ..., w^t_N) over N "strategies," normalizes it each round into an allocation, and after suffering a loss vector updates multiplicatively: "the weight vector wt is updated using the multiplicative rule." In the paper's own (OCR-corrupted) symbols this is w^{t+1}_i = w^t_i · β^{l^t_i}, which this book renders in clean notation as w_i^{t+1} = w_i^t \, \beta^{\,l_i^t}, where l_i^t \in [0,1] is strategy i's loss on round t and β \in [0,1] is a fixed parameter — strategies that lose more have their weight shrunk more aggressively. Section 3 of the paper specializes this scheme to boosting by treating each weak hypothesis as a "strategy" whose loss on an example is how badly it mispredicts that example, and treating each training example as a "trial." On each round t = 1, ..., T, "the booster devises a distribution Dt over the set of examples, and requests (from an unspecified oracle) a weak hypothesis (or rule-of-thumb) ht with low error =t with respect to Dt," and the resulting algorithm — Figure 2 in the paper, captioned "FIG. 2. The adaptive boosting algorithm" — instantiates Hedge's update rule example-by-example rather than strategy-by-strategy. Translating its steps out of the OCR-mangled Greek letters (β for their ";", ε for their "=", Σ for their run-together sums) into standard notation: given N labeled examples, initialize w_i^1 = D(i); on each round set p_i^t = w_i^t / \sum_j w_j^t, call the weak learner with distribution p^t to obtain h_t: X \to [0,1], compute its error \varepsilon_t = \sum_i p_i^t |h_t(x_i) - y_i|, set \beta_t = \varepsilon_t / (1-\varepsilon_t), and reweight w_i^{t+1} = w_i^t \, \beta_t^{\,1 - |h_t(x_i)-y_i|} — so an example h_t gets right (|h_t(x_i)-y_i| \approx 0) has its weight multiplied by roughly \beta_t, while an example h_t gets wrong keeps its weight almost unchanged, concentrating future rounds on the current mistakes exactly as Schapire's 1990 filtering argument required. "The hypothesis hf combines the outputs of the T weak hypotheses using a weighted majority vote," each hypothesis weighted by log(1/\beta_t) — the more accurate a round's weak hypothesis, the smaller \beta_t and the larger its vote. "We call the algorithm AdaBoost because, unlike previous algorithms, it adjusts adaptively to the errors of the weak hypotheses returned by WeakLearn." The paper proves that the training error of this combined hypothesis "is bounded by" an expression that, translated the same way, reads \exp\!\big(-2\sum_{t=1}^T \gamma_t^2\big) where \varepsilon_t = \tfrac12 - \gamma_t measures how far the t-th weak hypothesis clears random guessing — "this bound shows that if we can consistently find weak hypotheses that are slightly better than random guessing, then the error of the final hypothesis drops exponentially fast."
Breiman's derivation starts from an idealization it then admits is unavailable. Suppose the learner had, instead of one training set L, an infinite sequence of independent replicate training sets {L_k} drawn from the same distribution, each yielding a predictor \varphi(x, L_k). For numerical prediction the natural aggregate is the average over replicates, \varphi_A(x) = E_L[\varphi(x,L)]; for classification, the replicates vote and \varphi_A(x) is the plurality class. Since a real learner has only one L, Breiman substitutes bootstrap resampling for the missing replicates: draw {L^{(B)}} by sampling N cases with replacement from the single available L, refit on each, and average or vote as before. We call this procedure "bootstrap aggregating" and use the acronym bagging. The derivation of why this can only help — never strictly why it does, since that depends on the base learner — is an application of Jensen's inequality to squared error. Writing out E_L(y-\varphi(x,L))^2 for a fixed input x and expanding the square, the cross term E_L[y\,\varphi(x,L)] involves E_L[\varphi(x,L)] = \varphi_A(x) linearly, while the remaining term E_L[\varphi(x,L)^2] is bounded below by \varphi_A(x)^2 via E[Z^2] \geq (E[Z])^2. "Integrating both sides of (4.2) over the joint Y, x output-input distribution, we get that the mean-squared error of" the aggregated predictor "is lower than the mean-squared error averaged over" the replicates — an inequality with no free parameters and no assumption beyond independence of the replicates. The gap between the two sides is governed entirely by how much \varphi(x,L) varies across replicate training sets: a stable procedure (Breiman's running example is k-nearest-neighbor) has \varphi(x,L) nearly constant across L, so aggregation changes almost nothing, while an unstable one (trees, subset-selected linear regression, unregularized neural nets) has high replicate-to-replicate variance and gains correspondingly more. Classification breaks the unconditional guarantee. Let Q(j|x) be the fraction of replicates predicting class j at x and P(j|x) the true class posterior; Breiman calls the base predictor order-correct at x if its plurality vote across replicates agrees with the Bayes-optimal class — "this means that if input x results in class j more often than any other class, then" the base predictor "also predicts class j at x more often than any other class." Where the base learner is order-correct almost everywhere, aggregation drives its accuracy toward the Bayes rate; where it is not, "unlike the numerical prediction situation, poor predictors can be transformed into worse ones" by exactly the same voting mechanism. The proof for the numerical case has no analogue here because argmax, unlike the mean, does not satisfy Jensen's inequality in the needed direction — the classification guarantee is real but strictly weaker, and it can fail.
Schapire, Freund, Bartlett, and Lee's 1998 paper resolves the puzzle Section 1 left open: why AdaBoost's test error keeps falling once training error has already hit zero, apparently defying every complexity-penalized bound this book has derived since Chapter 6. Their diagnosis is that training error is the wrong statistic to watch once it reaches zero; the quantity still changing is the margin of a vote. For a classifier f built as a convex combination of base hypotheses drawn from a class H — exactly the weighted majority vote AdaBoost outputs after T rounds, with the normalized round-weights log(1/\beta_t) from Section 2's derivation playing the role of the combination coefficients — the margin of a labeled example (x,y) is the gap between the normalized vote cast for the correct label and the largest vote cast for any wrong one, margin(f,x,y) = f(x,y) - \max_{y' \neq y} f(x,y'); an example is misclassified exactly when this quantity is negative, and a positive-but-small margin means the vote barely went the right way. The paper's theorem bounds the true error P_D[\text{margin}(f,x,y) \le 0] not by the fraction of training examples with negative margin — which AdaBoost drives to zero and keeps there — but by the fraction whose margin fails to clear an arbitrary threshold \theta>0, plus a complexity term that depends on the VC-dimension d of the base class H, the sample size m, and \theta, and that — crucially — contains no dependence on T, the number of boosting rounds, or on how many base hypotheses have been combined into the vote. The version of the argument for a finite base class H is proved directly by a covering argument over the convex hull; the extension to infinite H of VC-dimension d reuses the same machinery, since, as the paper says of its own generalization, "The proof closely follows that of Theorem 1, so we only describe the differences" — the extra cost of an infinite base class enters only through d, never through T. This is exactly the resolution Section 1's puzzle requires: AdaBoost's exponential training-error bound (Section 2) keeps forcing the weighted vote to become more lopsided even once every training example is already on the correct side of zero, so margins keep growing through rounds that add nothing visible to the training-error curve — a fact the paper documents directly from its own experiments: "This can be seen in Figure 4, where we observe that the maximal margin in the final classifier is bounded well away from 1." Bagging has no mechanism that produces this effect, since it never reweights examples by how badly a previous round misclassified them, and its margin distributions look correspondingly different: "Contrast this with the margin distribution graphs after 1000 iterations of bagging in which as many as half of the examples have a margin of 1" — bagging's margins pile up at the maximum rather than concentrating gradually, consistent with Breiman's variance-reduction story (Section 2) rather than boosting's adaptive-reweighting one. The bound also predicts where boosting should fail: when the base hypotheses are too simple to be combined into large margins at all, no number of rounds helps, because "the complexity of the base classifiers, as explicitly indicated by Theorems 1 and 2. When used with stumps, boosting can achieve training error much smaller than that of the base learner; however, it is unable to achieve large margins" — the complexity term in the bound stays large regardless of T, because the achievable \theta is capped by the weak learner's own limitations rather than by how long boosting is run.
Breiman's 2001 random forests paper derives a third generalization guarantee for a committee, built to cover bagging and boosting-style voting within a single inequality rather than to explain either alone. Its object is the same as Section 3's margin bound — a vote over many classifiers — but the source of the diversity among voters is now an explicit random parameter rather than a bootstrap resample. A random forest is a collection of tree classifiers h(x,\Theta_k), one per tree, where the \Theta_k, drawn afresh for each tree, "are independent identically distributed random vectors," selecting at minimum which bootstrap resample and which random subset of input features that tree is permitted to split on; "each tree casts a unit vote for the most popular class at input x." Where this book renders the random parameter as \Theta, the paper's own printed symbol for it has not survived OCR and appears as a blank in the source text — the reader should take every bare "( , )" in the quotations below as standing for "(X,\Theta)" or "(\Theta,\Theta')". Breiman defines "the margin function for a random forest is mr(X, Y)" equal to the probability, over the distribution of \Theta, that the ensemble's vote goes to the true class Y minus the probability it goes to the best competing class: mr(X,Y) = P_\Theta(h(X,\Theta)=Y) - \max_{j\neq Y} P_\Theta(h(X,\Theta)=j). The strength of the ensemble is this margin's expectation over the data distribution, s = E_{X,Y}\,mr(X,Y) — "the strength of the set of classifiers" {h(x,\Theta)}. Chebyshev's inequality applied to the margin gives a first bound on the generalization error PE*, defined as the probability that mr is negative: assuming s\ge 0, "Chebychev's inequality gives" PE* \le \operatorname{var}(mr)/s^2. Breiman then bounds that variance from above using a pointwise object he calls the raw margin — "Definition 2.2. The raw margin function is rmg" (\Theta,X,Y) = I(h(X,\Theta)=Y) - I(h(X,\Theta)=\hat\jmath(X,Y)), where \hat\jmath(X,Y) is the class with the second-highest vote probability, so that mr(X,Y) is exactly the \Theta-expectation of the raw margin. Writing \bar\rho for the mean correlation between the raw margins produced by two independently drawn \Theta,\Theta', weighted by their standard deviations, and using the elementary fact that a difference of two indicator variables has variance at most one minus the square of its own mean, the variance of mr is bounded above by \bar\rho(1-s^2). Substituting into the Chebyshev bound gives the paper's central result: "An upper bound for the generalization error is given by" PE^* \le \bar\rho(1-s^2)/s^2. The bound names exactly two levers, and only two — the strength s of the individual trees, and the correlation \bar\rho between their raw margins across independent draws of \Theta — and it formalizes what bagging's variance argument (Section 2) could only gesture at: a committee generalizes well to the extent its members are each individually accurate and mutually decorrelated, and the two goals trade off against one another, since randomizing \Theta more aggressively to lower \bar\rho typically lowers s as well. Breiman names the ratio of the two effects directly: "The c/s2 ratio is the correlation divided by the square of the strength." "In understanding the functioning of random forests, this ratio will be a helpful guide" — the smaller it is, the better — and it is this ratio, not the training error and not the raw number of trees, that the random-feature-selection scheme derived in Section 3 is designed to push down.
Theorem 2.3's bound is a design principle stated as an inequality — minimize the c/s2 ratio — and Breiman's simplest algorithm is a direct attempt to act on it without ever estimating \bar\rho or s from a held-out set. Forest-RI, "the simplest random forest with random features," is formed "by selecting at random, at each node, a small group of input variables to split on. Grow the tree using CART methodology to maximum size and do not prune." "The size F of the group is fixed," and it is the one knob the algorithm exposes: a small F decorrelates the trees aggressively (lowering \bar\rho) at some cost to each tree's individual accuracy (lowering s), while F equal to the full input dimension recovers an ordinary bagged tree, correlated and strong. This is bagging (Section 2) with one further randomization layered on top — the bootstrap resample supplies one component of \Theta, the random feature subset supplies another — and the strength-correlation bound is what explains why the second layer of randomness, which looks like it should only hurt each individual tree, can still improve the vote. The same machinery that defines s and \bar\rho also estimates them without a test set, by reusing trees a bootstrap resample never saw: Breiman aggregates, for each training case, only the votes of the trees whose bootstrap sample excluded it — "Call this the out-of-bag classifier. Then the out-of-bag estimate for the generalization error is the error rate of the out-of-bag classifier on the training set" — an internal, honest proxy for PE* obtained from the training data alone, because every case is scored only by trees that never trained on it. Breiman reports the practical payoff of trading adaptive reweighting (boosting, Section 2) for randomized feature selection: on the same benchmarks, "growing random forests is many times faster than growing the trees based on all inputs needed in Adaboost," and concretely, "a Forest-RI run (F = 1) on the zip-code data takes 4.0 minutes on a 250 Mhz Macintosh to generate 100 trees compared to almost three hours for Adaboost." That specific ratio is a 1999 workstation's arithmetic and would collapse on any machine built since; what does not collapse is the structural reason the two committees differ in cost at all — AdaBoost's rounds are serial by construction, each weak learner trained on a distribution that depends on every previous round's errors, while a random forest's trees are independent draws of \Theta that can be grown in any order, or, on a machine Breiman did not have, all at once.
Section 2 derived AdaBoost's exponential training-error bound and the Schapire-Freund-Bartlett-Lee margin bound that explains what happens after training error hits zero; this section reruns both phenomena on hardware and data no 1996-1998 experiment could use, and checks whether the qualitative claims survive. The rerun implements discrete AdaBoost exactly as derived in Section 2.b1, not sklearn's default real-valued SAMME.R variant: weights w_i^1 = 1/N; at each round a decision stump (max_depth=1, the weakest possible tree, matching the "weak hypothesis" role in the derivation) is fit with sample weights w^t; its weighted error \varepsilon_t against 0/1 labels gives \beta_t = \varepsilon_t/(1-\varepsilon_t) and vote-weight \alpha_t=\log(1/\beta_t); every example's weight is multiplied by \beta_t^{1-|h_t(x_i)-y_i|} and renormalized, exactly the update in Section 2.b1. The dataset is scikit-learn's digits (8x8 handwritten-digit bitmaps, the modern bundled descendant of the zip-code recognition problem both Freund-Schapire and Breiman used as a benchmark), restricted to the classically confusable pair 3-vs-8 to keep the binary derivation exact rather than reaching for a multiclass extension neither paper's core theorem covers. 232 training examples, 125 held out for test, 300 boosting rounds. This is a substitution the derivation does not license silently: the corpus's 1998 experiments used NIST/USPS zip-code scans an order of magnitude larger, and this rerun's numbers should be read as a check on the qualitative shape of the claims — training error to zero, margin still growing, generalization not degrading — not as a replication of the original paper's reported error rates.
The training/test error trajectory reproduces the qualitative shape Section 2.b3 built its argument around. Training error falls from 0.1422 at round 1 to 0.0043 at round 10 to exactly 0.0000 at round 20, and stays at 0.0000 through round 300 — 280 further rounds of boosting that add nothing visible to the training-error curve. Test error over the same span falls from 0.0960 (round 1) to 0.0480 (round 10) to 0.0240 (round 20), and from there wanders in a narrow band — 0.0320 at round 50, back down to 0.0240 by round 75 and remaining there through round 300 — rather than climbing back up as a training-error-only account of generalization would predict once the model has "memorized" the training set twenty rounds in. That flat-not-rising test error is the qualitative claim; what the margin bound says should be happening underneath it is checked directly by tracking the training-set margin distribution, margin(f,x,y) = f(x,y)-f(x,\bar y) normalized so f's coefficients sum to one, at successive rounds after training error has already reached zero. The minimum training margin — the single hardest-to-classify example, and the quantity the covering-number bound in Section 2.b3 actually controls through the threshold \theta — rises from 0.090 at round 20 to 0.199 at round 50 to 0.239 by round 250, more than doubling while the training-error curve is dead flat at zero throughout. The bulk of the distribution does not move monotonically in the same direction: the median margin actually falls from 0.509 (round 20) to 0.361 (round 250) as boosting keeps reweighting the still-hardest examples at the expense of ones that were already comfortably classified, and correspondingly the fraction of training points with margin below 0.5 rises from 0.474 to 0.858 over the same rounds — a real finding, not the monotone-everywhere growth a loose paraphrase of "boosting increases the margins" might suggest, and consistent with the original paper's own qualifier that boosting is "willing" to trade large margins on easy examples for gains on hard ones. What is unambiguous is the minimum: it only grows, and it is the minimum, not the median, that Theorem 2's bound is written in terms of.
Section 2.b4-b5 derived Breiman's bound PE^* \le \bar\rho(1-s^2)/s^2 and named F, the number of candidate features considered at each split, as the one knob Forest-RI exposes for trading strength against correlation. This rerun estimates s and \bar\rho directly from a fitted forest rather than taking the bound on faith: on the full ten-class digits task (64 pixel features, 1257 training / 540 test examples), for each tree in a 200-tree Forest-RI-style random forest the per-example raw margin rmg(\Theta,x,y) = I(h(x,\Theta)=y) - I(h(x,\Theta)=\hat\jmath(x,y)) is computed exactly as defined in Section 2.b4, with \hat\jmath(x,y) taken as the class receiving the most votes among trees after excluding the true class; averaging rmg over trees gives mr(x,y) and hence the empirical strength s = \text{mean}_{x,y}\,mr(x,y); the empirical \bar\rho is the mean pairwise correlation of the raw-margin vectors across the 200 trees. Sweeping F over {1,2,4,8,16,32,64} (F=64 uses every feature at every split, i.e. ordinary bagging with no feature randomization) gives: at F=1, s=0.510, \bar\rho=0.126, bound=0.359, actual test error=0.0352; at F=8, s=0.663, \bar\rho=0.169, bound=0.216, actual test error=0.0222; at F=16, s=0.699, \bar\rho=0.195, bound=0.205, actual test error=0.0222; at F=32, s=0.719, \bar\rho=0.210, bound=0.196, actual test error=0.0259; at F=64 (bagging), s=0.738, \bar\rho=0.277, bound=0.232, actual test error=0.0407. Both the bound and the true error are U-shaped in F rather than monotone, exactly as Theorem 2.3 predicts they should be: shrinking F below its optimum decorrelates trees faster than it weakens them, so both quantities fall; past the optimum, correlation climbs faster than strength compensates, so both rise again. The bound's minimum (0.196-0.205, F between 16 and 32) and the actual test error's minimum (0.0222, F between 8 and 16) sit in the same neighborhood of F rather than at the extremes — full-feature bagging at F=64 is neither the best nor the worst choice on this data, but it is measurably worse than several intermediate F, which is the concrete, checkable content of "the c/s^2 ratio is the correlation divided by the square of the strength" and "the smaller it is, the better" from Section 2.b4.
Section 2.b5 closed on Breiman's own timing comparison — 4.0 minutes for Forest-RI versus almost three hours for AdaBoost on a 250 MHz Macintosh — and the claim made there was that the specific ratio is 1999 hardware arithmetic while the structural reason for it (AdaBoost's rounds are serial, a random forest's trees are independent draws of \Theta) is not. This is directly checkable on present hardware, where sub-second training times make wall-clock minutes moot but a machine's core count is available to test the structural claim instead of the timing one. On a 15-core machine, training 200 trees for a random forest on the digits task takes 0.203 seconds restricted to a single core (n_jobs=1) and 0.131 seconds allowed to use every core (n_jobs=-1) — a 1.55x speedup, submaximal because per-tree fitting on a dataset this small is dominated by fixed overhead rather than by CPU-bound splitting, but a real, measured speedup from parallelism nonetheless, because the 200 trees are independent draws of \Theta and can be assigned to cores in any order. Training 200 rounds of discrete AdaBoost with depth-1 stumps on the same ten-class task (via sklearn's AdaBoostClassifier, which exposes no n_jobs parameter for exactly the reason Section 2.b5 gives) takes 0.263 seconds — comparable in absolute terms on this small dataset, where nothing takes long enough for the structural difference to show up as a wall-clock gap the way it did on Breiman's zip-code-sized problem, but categorically different in whether more cores can help at all: there is no parameter to ask sklearn's AdaBoost implementation to use more than one core, because round t+1's training distribution is defined only after round t's weak hypothesis has been fit and scored, so no reordering of the 200 rounds across cores preserves the algorithm. A 2026 datacenter with a thousand-fold more cores than this rerun's fifteen would shrink the random forest's wall-clock time by roughly another order of magnitude and would not shrink AdaBoost's serial critical path by any amount at all beyond what a faster single core buys. That is the finding Breiman's paper actually supports once the specific minutes are discounted: not that boosting is slow and forests are fast, but that one algorithm's cost scales with the number of available cores and the other's does not, a distinction hardware improvements move further apart rather than erasing.
Reinforcement learning inherited its formal apparatus from two lineages that did not at first look like the same problem. One is the prediction problem that opens Richard Sutton's 1988 paper: "the problem of learning to predict, that is, of using past experience with an incompletely known system to predict its future behavior." Sutton's own setting is deliberately austere. Experience arrives as a sequence with a single terminal signal rather than a reward at every step — "experience comes in observation-outcome sequences of the form x1, x2, x3,..., xm, z, where each xt is a vector of observations available at time t in the sequence, and z is the outcome of the sequence" — and the learner's job is to produce, at each time t, a prediction Pt of that eventual z, updating it as new observations arrive rather than waiting for the sequence to end. This is a prediction problem with no notion yet of an agent choosing among actions, no reward at every tick, and no discounting: it is the mathematics of forecasting an outcome from a Markov chain the learner does not control. The other lineage is the control problem formalized by Bellman and inherited by Chris Watkins and Peter Dayan in their 1992 paper on Q-learning: an agent that acts in a world at every step, receives a reward for each action, and must choose actions to maximize the sum of rewards it can expect to receive from now on, discounted so that a reward available sooner is worth more than the same reward available later. What the rest of this chapter has to explain is how these two objects — a prediction updated from a stream of temporally successive guesses, and a decision problem defined by an accumulated, discounted reward — turn out to be the same mathematics viewed from two directions, and how the fixed-point equation named for Bellman is the hinge between them.
Watkins and Dayan set out the control formalism in the notation this chapter will use throughout. "Consider a computational agent moving around some discrete, finite world, choosing one from a finite collection of actions at every time step. The world constitutes a controlled Markov process with the agent as a controller." At step n the agent observes a state x_n drawn from a finite set X, chooses an action a_n from a finite set of actions, and the world responds in two ways at once: a probabilistic reward r_n, whose mean value R_x(a) depends only on the current state and the action taken, and a transition to a new state y_n according to a fixed law P_xy[a] — the probability of landing in y given that action a was taken in x. Nothing here has yet been learned; R_x(a) and P_xy[a] are properties of the world, not of the agent, and the Markov assumption is doing real work: the reward and the next state depend on x_n and a_n alone, not on the path that led there. The return the agent is trying to maximize is defined by discounting: "The task facing the agent is that of determining an optimal policy, one that maximizes total discounted expected reward. By discounted reward, we mean that rewards received s steps hence are worth less than rewards received now, by a factor of γ^s (0 < γ < 1)." A policy π is a rule assigning an action to every state, and the value of a state under a fixed policy π folds the immediate reward and the discounted value of what follows into a single recursive quantity: V^π(x) is defined so that, in Watkins and Dayan's words, "the agent expects to receive R_x(π(x)) immediately for performing the action π recommends, and then moves to a state that is 'worth' V^π(y) to it, with probability P_xy[π(x)]" — that is, V^π(x) = R_x(π(x)) + γ Σ_y P_xy[π(x)] V^π(y), a one-step average of immediate reward plus discounted future value, self-referential because V^π appears on both sides. (The displayed equation itself did not survive extraction from the source PDF; the equation given here is reconstructed from the surrounding sentence, which states its content unambiguously, and follows Watkins and Dayan's own symbols X, Y, R, P, π, γ throughout.)
Fixing a policy π gives a linear recursion for V^π, but the object the agent actually wants is the best attainable value at every state, independent of any one policy. Call this V*(x). The corresponding recursion replaces "the reward for the action π recommends" with the reward for whichever action does best, and replaces "the value of the state π leads to" with the best-attainable value of that next state: V*(x) = max_a [R_x(a) + γ Σ_y P_xy[a] V*(y)]. (As with the fixed-policy recursion in the previous paragraph, the displayed equation itself did not survive extraction from the source PDF — Watkins and Dayan's text jumps straight from "the value of state x is" to the explanatory sentence that follows, with the equation lost in between. The form given here is reconstructed from that surrounding prose and is the standard statement of Bellman's optimality equation for a discounted Markov decision process, in the paper's own X, Y, R, P, γ notation.) The max inside the sum is Bellman's principle of optimality: the best value of a state is the best immediate reward plus the best value of wherever that action leads, not the value of committing in advance to one fixed policy. Because V* appears on both sides of its own defining equation, the definition can look circular, and Watkins and Dayan raise the worry only to dismiss it: "The theory of DP (Bellman & Dreyfus, 1962; Ross, 1983) assures us that" there "is at least one optimal stationary policy T* which is such that" it "is as well as an agent can do from state x." Read past the OCR damage — "T*" is π*, the optimal policy, one of several places where the extraction has substituted the Latin capital T for the Greek π — and the claim is the standard dynamic-programming existence result: the fixed point exists, and at least one stationary policy attains it. Granting existence, the same theory is constructive: the equation is "well defined, and DP provides a number of methods for calculating V*" and π*, provided that "(Rx(a) and Pxy[a] are known" in advance. Watkins and Dayan then introduce the quantity the rest of the chapter turns on: define Q*(x, a) as the discounted return of taking action a in state x and behaving optimally forever after. The link back to V* is immediate — "It is straightforward to show that V*(x) = maxa Q*(x, a)," the state value being just the value of its best action — and reading off behavior is immediate too, since "an optimal policy can be formed as r*(x) = a*," where the OCR renders π*(x) as "r*(x)" and the maximizing action as a*. The move from V* to Q* looks like bookkeeping, but it is the whole point of what follows: computing the max in V*'s own equation requires already knowing R_x(a) and P_xy[a], while Q*, once known, lets the agent read off the optimal action by comparison alone, without ever representing the transition law that produced it.
Everything in the previous paragraph is dynamic programming in the classical sense: it presumes R_x(a) and P_xy[a] are given, and computes V*, π*, or Q* from them by whatever DP method is convenient. That is not the situation of an agent dropped into an unfamiliar world. As Watkins and Dayan put it, "The task facing a Q, learner is that of determining" — the OCR drops a symbol here; the sentence names the optimal policy — "without initially knowing these values": R_x(a) and P_xy[a] are properties of a world the agent has not yet mapped. One traditional response, which the paper credits to Sato, Abe and Takeda (1988), is to learn R_x(a) and P_xy[a] from experience and run DP on the estimates as they improve — the certainty-equivalence approach, substituting the current model for the true one and acting as if the substitution were free. Watkins and Dayan's objection to this is not that it fails asymptotically but that it fails early, when it matters most for an agent that has to act while it is still learning: "any assumption of certainty equivalence" — "calculating actions as if the current model" is accurate — "costs dearly in the early stages of learning." Q-learning is offered as the alternative that sidesteps the estimation step entirely: instead of learning R_x(a) and P_xy[a] and then solving Bellman's equation for them, the agent learns Q*(x, a) directly, incrementally, from the reward and next-state samples its own actions produce, and the guarantee the rest of this chapter derives is that this direct, model-free update converges to the same Q* that classical DP would have computed from a perfectly known model. Sutton's temporal-difference method, introduced from the other lineage in the opening paragraph of this section, is the tool that makes an update of that kind possible: an estimate revised from the next estimate rather than only from a final outcome. The next section derives that revision rule, first for prediction without control, and then, via Watkins and Dayan's convergence proof, for Q-learning itself.
Sutton's derivation starts from the supervised-learning update it means to displace. In his notation, a prediction sequence P1, P2, ..., Pm is produced for a single observation-outcome sequence x1, ..., xm, z, each Pt a function of xt and a weight vector w. The conventional rule waits for z and moves every weight by the gradient of every past prediction toward it: Δw_t = α(z − P_t)∇_w P_t, "where α is a positive parameter affecting the rate of learning, and the gradient, ∇_wP_t, is the vector of partial derivatives of P_t with respect to each component of w." Applied with a linear predictor this is exactly the Widrow-Hoff delta rule, and it cannot be computed incrementally: every Δw_t depends on z, so nothing can be updated until the sequence ends. Sutton's move is an algebraic identity, not a new assumption: the terminal error telescopes into a sum of successive differences, z − P_t = Σ_{k=t}^{m}(P_{k+1} − P_t), with P_{m+1} defined as z itself. Substituting this sum into the supervised update and regrouping by the earlier prediction being corrected converts the batch rule into a rule that can be evaluated one step at a time — "the key is to represent the error z − P_t as a sum of changes in predictions." The result, which Sutton calls TD(1), moves each past prediction by the very next prediction's change: Δw_t = α(P_{t+1} − P_t)Σ_{k=1}^{t}∇_wP_k, and he proves the two are identical over a whole sequence — "On multi-step prediction problems, the linear TD(1) procedure produces the same per-sequence weight changes as the Widrow-Hoff procedure" (Theorem 1). Nothing has been changed about what the method learns; only when the arithmetic happens.
TD(1) treats every earlier prediction as equally responsible for the newest error, which is the sense in which it still agrees with the batch method. Sutton then asks what happens if more recent predictions are blamed more heavily, weighting the correction to the prediction k steps in the past by λ^k for a fixed 0 < λ < 1 (Sutton's own symbol is rendered "A" by the OCR throughout this passage of the source, a Latin capital substituted for the Greek λ; the surrounding text is unambiguous about which is meant). This gives the TD(λ) family, and Sutton names it directly: "Accordingly, we call this new procedure TD(A) and we will refer to the procedure given by (3) as TD(1)." Because the weighting is exponential, it can still be accumulated incrementally in a single vector e_t — an eligibility trace — updated as e_{t+1} = ∇_wP_{t+1} + λe_t, so that no history of past gradients need be stored. At the other extreme from TD(1) sits TD(0), λ = 0, which discards everything but the very last prediction: "the weight increment is determined only by its effect on the prediction associated with the most recent observation," Δw_t = α(P_{t+1} − P_t)∇_wP_t. This is the rule the rest of the chapter needs. Read against the Bellman equation of the previous section, P_t is playing the role of an estimated value V(x_t), and P_{t+1} − P_t is the one-step prediction error that a Bellman-style consistency condition says should be zero at the fixed point — the discounted-reward general case simply inserts a reward and a discount factor into that same difference, r_t + γP_{t+1} − P_t, of which Sutton's undiscounted, reward-only-at-termination setting is the special case with γ = 1 and r_t = 0 for all nonterminal t. TD(0) is thus gradient descent not on the distance between a prediction and the true outcome — which is unknown until the sequence ends — but on the distance between a prediction and the very next prediction, the quantity Watkins and Dayan's Q-learning update will reuse verbatim.
Having shown TD(1) matches Widrow-Hoff, Sutton asks the harder question: does the genuinely different rule, TD(0), still converge to something correct? The setting is an absorbing Markov process, "in which each next state depends only on the current state," defined by nonterminal states N, terminal states T, and transition probabilities p_ij; the "absorbing" property means that indefinite cycles among the nonterminal states are not possible, and all sequences eventually terminate. A sequence q1, ..., q_{m+1} is generated by starting from some nonterminal state drawn from a fixed distribution and following the p_ij until absorption, with an outcome z drawn from a distribution attached to the terminating state. Against this generative model Sutton proves: "For any absorbing Markov chain, for any distribution of starting probabilities u2, for any outcome distributions with finite expected values Zj, and for any linearly independent set of observation vectors {xt | i € N}, there exists an e > 0 such that, for all positive a < e and for any initial weight vector, the predictions of linear TD(0) (with weight updates after each sequence) converge in expected value to the ideal predictions" (Theorem 2; the OCR renders μ_i as "u2," ε as "e," and α as "a" throughout). "Ideal" here means the true expected outcome conditional on the current state, exactly the quantity a converged value function V(x) would have to equal at the fixed point of the Bellman recursion from the previous section. The proof groups the per-sequence weight change by which state-to-state transition produced it, so that the increment becomes a sum over pairs i, j of n_ij, the number of times the transition i → j occurred in the sequence, times a term that vanishes exactly when the weights already give the correct expected outcomes; taking expectations over sequences turns n_ij into d_i p_ij, where d_i is the expected number of visits to state i per sequence, and the fixed point of this expected update is shown, using the absorbing chain's fundamental matrix, to be the true expected outcomes. The two hypotheses doing the real work are linear independence of the observation vectors — so that no weight update solving the equations for one state distorts another — and a learning rate small enough relative to a bound the proof constructs from the chain's transition structure; neither is a statement about a machine, only about the geometry of the prediction problem and the size of α.
Watkins and Dayan's Q-learning takes the TD(0) error and points it at Q rather than V. In their notation, the agent's experience is a sequence of episodes; "in the nth episode, the agent observes its current state xn, selects and performs an action an, observes the subsequent state yn, receives an immediate payoff rn, and adjusts its Qn-1 values using a learning factor an," according to Q_n(x_n,a_n) = (1 − α_n)Q_{n-1}(x_n,a_n) + α_n[r_n + γ V_{n-1}(y_n)], updating only the one state-action pair visited and leaving every other entry of the table unchanged, where V_{n-1}(y) = max_b Q_{n-1}(y,b) "is the best the agent thinks it can do from state y." Compare this to TD(0): the bracketed quantity r_n + γV_{n-1}(y_n) is exactly Sutton's next prediction P_{t+1}, now built from a real reward plus a discounted bootstrap off the agent's own table rather than off a separate prediction sequence, and the update again moves the current estimate a fraction of the way toward it. The paper is explicit that this requires "a look-up table representation for the Q_n(x,a)" — "Watkins (1989) shows that Q-learning may not converge correctly for other representations" — and that the "most important condition implicit in the convergence theorem" is that "the sequence of episodes that forms the basis of learning must include an infinite number of episodes for each starting state and action." Under those conditions, and writing n_i(x,a) for the index of the i-th time action a is tried in state x, the convergence theorem states: "Given bounded rewards |r_n| < R, learning rates 0 ≤ α_n < 1," and the standard stochastic-approximation conditions on those rates — the paper's own display of Σα_{n_i(x,a)} = ∞ and Σα_{n_i(x,a)}^2 < ∞ did not survive extraction, but the surrounding appendix confirms it is the Robbins-Monro condition, decaying step sizes that shrink fast enough to control noise yet slowly enough that no state-action pair is ever undercorrected — "then Q_n(x,a) → Q*(x,a) as n → ∞, ∀x,a, with probability 1." This is the same conclusion Section 1's Bellman equation was building toward: the fixed point computable in advance by dynamic programming, when R_x(a) and P_xy[a] are known, is reachable instead by a purely local, model-free update from experience alone.
Two lemmas carry the argument. Lemma A ties the artificial process back to the real Q-values effectively by construction: "Qn(x, a) are the optimal action values for ARP states (x, n) and ARP actions a," proved by backward induction down the stack. Lemma B does the substantive work, in four steps the paper labels B.1 through B.4. B.1 is a discount-factor bound: for a discounted, bounded-reward finite Markov process, "the difference between the value of that state under the finite sequence of s actions and its value under that same sequence followed by any other actions tends to 0 as s → ∞," because the (s+1)th state is weighted by γ^s, which vanishes. B.2 is a level-escape bound: "given any level l, there exists another yet higher level, h, such that the probability can be made arbitrarily small of straying below l after taking s actions in the ARP, starting from above h" — so a card deck entered high enough in the stack is overwhelmingly likely to stay above any fixed level for as many actions as the argument needs. B.3 is where the learning-rate conditions in the theorem actually do their only work: "with probability 1, the probabilities P and expected rewards R in the ARP converge... to the transition matrices and expected rewards in the real process as the level n increases to infinity," because "since its raw data are unbiased, the conditions on the sums and sums of squares of the learning rates... ensure the convergence with probability one." B.4 is a continuity bound converting closeness of transition probabilities and rewards into closeness of values: "the discrepancy in the action values over a finite number s of actions between the values of two approximately equal Markov processes grows at most quadratically with s," so close probabilities and rewards force close values. Chained together — discount bound, escape bound, empirical convergence, continuity — these give the paper's conclusion, and note what the chain does not require: no mention of a machine, a clock, or a memory budget appears anywhere in it, only the deck construction, the discount factor, and the Robbins-Monro-style condition on the learning rates. "So, with probability 1, Qn(x, a) - Q*(x, a) as n - o as required."
The theorem Watkins and Dayan prove is narrower than the algorithm Watkins had already described in 1989, and the paper is careful to say so. What the action-replay-process argument licenses is one-step, one-update Q-learning: a single table entry corrected once per episode from a single bootstrapped target. Two extensions are sketched rather than proved in full. Updating several Q-values per episode instead of one goes through with "no non-trivial modification," since the same action-replay deck can be re-entered at any level at which a state-action pair actually occurred. The undiscounted, absorbing-goal-state case also goes through, because the certainty of eventual trapping does the work that the discount factor γ<1 did in the original proof — it bounds the difference between the value of finitely many actions and the value of infinitely many, which is all Lemma B.1 required. What does not go through is the case the previous derivation in this section spent most of its effort on: "The theorem above only proves the convergence of a restricted version of Watkins' (1989) comprehensive Q-learning algorithm, since it does not permit updates based on the rewards from more than one iteration. This addition was pioneered by Sutton (1984; 1988) in his TD(X) algorithm, in which a reward from a step taken r iterations previously is weighted by Xr, where X < 1." (The paper's "X" is again the OCR's substitute for λ.) Eligibility traces, the very mechanism that made TD(λ) able to interpolate between Widrow-Hoff and TD(0), sit outside what the action-replay process can certify: "Unfortunately, the theorem does not extend trivially to this case, and alternative proof methods such as those in Kushner and Clark (1978) may be required." Q(λ)'s convergence proof was therefore not a corollary of this one: the paper points to Kushner and Clark's stochastic-approximation machinery as the likely route to such a proof, but does not carry it out, and no such extension is derived here. None of these gaps are gaps a faster machine could close. The action-replay process is a purely combinatorial device — a deck of cards, a biased coin, a level in a stack — and every hypothesis it needs (bounded rewards, a Robbins-Monro schedule on the learning rates, infinitely many visits to each state-action pair) is a statement about the sampling process and the representation, never about clock time. The paper's own summary of what it has shown makes the same point from the other direction: "Such a guarantee has previously eluded most methods of reinforcement learning." What eluded them was a proof, not a processor. And the boundary the paper does draw — a look-up table only, one update per episode only, no eligibility trace — is exactly where the guarantee stops regardless of how many episodes a 2026 cluster could generate in an afternoon; extending it to a compressed, generalizing representation is the unresolved mathematical problem that reappears, decades later, whenever a value function is a neural network rather than a table.
Two claims from the previous section are checkable without archival hardware: Sutton's empirical comparison of TD(λ) procedures on a bounded random walk, and Watkins and Dayan's convergence guarantee for tabular Q-learning. Neither experiment strains a laptop. Sutton's random-walk task has five nonterminal states and its largest run presented a hundred training sets of ten sequences each; Watkins and Dayan's proof concerns a lookup table, not a network, and needs no more than a few thousand simulated episodes to see the guaranteed convergence, and its dependence on the learning-rate schedule, take shape. That is itself a data point for this book's recurring question. The 1988 paper's central results — that TD(1) exactly reproduces Widrow-Hoff, that TD(0) converges to the correct conditional expectations, that intermediate λ can generalize better than either extreme from a small sample — are claims about an algorithm's mathematics and about generalization from limited data, not about what a Reagan-era workstation could finish overnight. The reruns below reproduce both experiments in numpy on modern hardware, in seconds, and ask two things: does the qualitative finding survive, and if it does not survive exactly, is the discrepancy about data or about a clock.
Sutton's testbed is the bounded random walk of his Figure 2: seven states A through G in a line, a walk that always begins in the center state D and at every step moves to a neighboring state with equal probability, terminating the moment it enters A or G — "A typical walk might be DCDEFG." The five nonterminal states B, C, D, E, F are represented as unit basis vectors in R^5 — "the vectors {xi} were the unit basis vectors of length 5" — so each weight component is, in effect, a separate estimate of one state's chance of a right-side termination, and because the walk is symmetric those chances are known exactly: ⅙, 2/6, 3/6, 4/6, 5/6 for B through F, the target this rerun calls the ideal predictions. The rerun below reproduces two training regimes Sutton ran against that target. In the first, repeated-presentations regime, "each training set was presented repeatedly to each learning procedure until the procedure no longer produced any significant changes in the weight vector," accumulating Δw over an entire ten-sequence set before applying it, "and always to the same final value, independent of its initial value" — Sutton calls this "the repeated presentations training paradigm," and the reported error is "averaged over 100 training sets." In the second, single-presentation regime, each training set is shown to the learner only once rather than to convergence, and "weight updates were performed after each sequence, as in (1), rather than after each complete training set," swept over "a range of values for the learning-rate parameter a," with weights in both regimes initially set to 0.5 "so that there was no bias either toward right-side or left-side terminations."
Reproducing the repeated-presentations regime — one hundred training sets of ten sequences each, feature vectors and initial weights as above, weight changes accumulated over a full training set and the set re-presented until the changes fall below a small tolerance — gives, for λ ∈ {0, 0.1, 0.3, 0.5, 0.7, 0.9, 1}, RMS errors against the ideal predictions of 0.128, 0.128, 0.130, 0.132, 0.135, 0.145, and 0.173 respectively (run 248-sutton-random-walk-batch.py). The ordering matches Sutton's Figure 3 exactly: error increases monotonically with λ, TD(0) is the best performer, and Widrow-Hoff — λ = 1, mathematically identical to TD(1) by Theorem 1 of the previous section — is the worst by a wide margin, more than 30% worse than TD(0) in RMS terms in this rerun. Sutton reports this same gap only graphically, in Figure 3, and describes it in prose as TD(1) producing "the worst estimates" of any λ; the paper states no numeric RMS values, so the magnitude of the rerun's gap cannot be checked against a reported number — only the ordering can, and there it matches exactly. The reason is the one the previous section's derivation predicts: in the repeated-presentations paradigm the batch method converges to the ordinary least-squares fit of observed sequences to outcomes, which is optimal for the training data actually seen but is not the same target as the true conditional expectations TD(0) is proven to reach; with only ten sequences per set the two targets diverge, and the batch method pays for fitting the sample. Nothing in the widening gap depends on how fast the arithmetic runs — the same ordering appears whether the hundred training sets take an afternoon on 1988 equipment or a fraction of a second in the rerun.
Reproducing the second, single-presentation regime is where Sutton's own result is easy to misstate. Here each of the hundred training sets is shown once, weights are updated after every sequence rather than after every set, and the learning rate α is swept over a range of values for each λ, with the best α for each λ selected after the fact — Sutton reports that "the value of a had a significant effect on performance, with best results obtained with intermediate values," and that "for all values, however, the Widrow-Hoff procedure, TD(1), produced the worst estimates," so that "all of the TD methods with A < 1 performed better both in absolute terms and over a wider range of a values than did the supervised-learning method." The rerun matches this at every value of λ tested (run 249-sutton-random-walk-online.py): sweeping α ∈ {0.05, ..., 0.20} and reporting each λ's best-α error gives, for λ ∈ {0, 0.3, 0.5, 0.8, 1}, RMS errors of 0.091 (α = 0.20), 0.088 (α = 0.20), 0.094 (α = 0.15), 0.120 (α = 0.10), and 0.188 (α = 0.05) — Widrow-Hoff again the clear worst, exactly as reported. The value worth stating precisely, because it is easy to round off to "TD(0) wins" from memory of the batch result two paragraphs above: it does not win here. Sutton's Figure 5, plotting each λ's best-α error, states the finding exactly — "as in the repeated-presentation experiment, all A values less than 1 were superior to the A = 1 case. However, in this experiment the best A value was not 0, but somewhere near 0.3" — and the rerun's own minimum, 0.088 at λ = 0.3, lands at exactly the value Sutton names, ahead of TD(0)'s 0.091.
Watkins and Dayan's paper contains no experiment to rerun — it is a proof, not a simulation study, and the previous section already quoted its theorem in full. What it does supply is a short list of hypotheses the guarantee needs — the note that "this description assumes a look-up table representation for the Q,n(x, a)" (the comma is the OCR's rendering of the subscript n), bounded rewards, and, in the paper's own words, "the most important condition implicit in the convergence theorem given below is that the sequence of episodes that forms the basis of learning must include an infinite number of episodes for each starting state and action," together with a Robbins-Monro schedule on the learning rates — and the point of building an experiment here is to watch what happens exactly at the boundary of that list, where the theorem stops applying. The rerun uses a five-state chain: states 0 through 4, two actions (step left or step right), each action succeeding with probability 0.8 and slipping to the opposite neighbor with probability 0.2, reward 1 for the transition that leaves state 4 to the right, reward 0 everywhere else, discount γ = 0.9, and absorption the instant either end of the chain is left. Because the model is fully specified, Q* is computable directly by value iteration on the Bellman optimality equation of the previous chapter's first section, giving a ground truth to check the learned table against rather than only a claim of eventual agreement. A tabular Q-learner is then run against this chain with the update Watkins and Dayan give, Q_n(x_n,a_n) = (1 − α_n)Q_{n-1}(x_n,a_n) + α_n[r_n + γV_{n-1}(y_n)], actions chosen uniformly at random so that every state-action pair is visited infinitely often as the theorem requires, under two learning-rate schedules: a decaying α_n(x,a) = 1/n(x,a), the classic Robbins-Monro choice for which Σα diverges and Σα² converges, and a constant α = 0.3, which satisfies the first sum but not the second.
Value iteration on the known model (run 252-watkins-qlearning-rerun.py) gives the ground truth against which both schedules are checked: Q*(0,·) = (0.101, 0.404), Q*(1,·) = (0.413, 0.561), Q*(2,·) = (0.549, 0.679), Q*(3,·) = (0.659, 0.802), Q*(4,·) = (0.778, 0.944) — increasing left to right and top to bottom, as it should for a chain whose only reward sits past state 4. Under the decaying schedule α_n = 1/n(x,a), the maximum entrywise error between the learned table and Q* falls steadily as episodes accumulate: 0.173 after 1,000 episodes, 0.104 after 5,000, 0.064 after 20,000, 0.045 after 50,000 — not perfectly monotonic run to run, since this is a stochastic process, but converging in the sense the theorem promises, error shrinking toward zero with no floor in sight. Under the constant schedule α = 0.3, three independent runs of 50,000 episodes each — same chain, same random exploration, same number of updates, different random seeds — land at final errors of 0.085, 0.145, and 0.203. The table is not still improving at 50,000 episodes the way the decaying-rate table was; it has stopped improving, and where it stops is a matter of which noise sequence the seed happened to produce, not of how long the run continues. That is precisely the distinction Lemma B.3 of the previous section draws: with Σα² < ∞, the accumulated noise in the estimate is summable and the process settles on the true value; with Σα² = ∞, as for any constant α > 0, the noise keeps arriving at a rate that never decays, so the process wanders in a ball around Q* whose size depends on α and never fully closes. More episodes at constant α do not fix this — running the constant-α case for another 50,000 episodes moves the table around within the same noise ball, it does not shrink the ball, which is the qualitative signature the theorem's hypothesis, not the theorem's guarantee, would predict for a schedule that violates it.
Put the two reruns side by side and a single pattern emerges. Sutton's random walk reproduces exactly, in both regimes, on a laptop in under a second per condition — a hundred training sets of ten five-state sequences was never a computational challenge, on 1988 hardware or any other, and the rerun's only discrepancy with a casual restatement of the paper is a discrepancy in memory, not in mathematics: it is easy to remember "TD(0) wins" from the batch experiment and misapply it to the online one, where Sutton's own Figure 5 and this rerun agree that the winner is λ ≈ 0.3. The reason is exactly the one the derivation in the previous section gives: with only ten single-pass sequences per set, TD(0)'s bias toward the correct conditional expectation is not enough to beat a method that trades a little bias for faster propagation of information along the sequence, a bias-variance tradeoff visible only because the training set is small — give either method enough repeated presentations of the same data and the batch result of the first rerun shows TD(0) pulling back ahead, because then it is fitting the same distribution the online experiment could only sample from once. Watkins and Dayan's convergence theorem needed no reproduction of a numerical result at all, because the paper reports none; the exercise instead was to build the small experiment the theorem implies and watch its own hypotheses matter. A five-state chain and a lookup table are not a strain on any decade's hardware, and the difference between the decaying and constant learning-rate runs — one settling, the other wandering in a fixed-size ball no matter how many more episodes are added — is not a difference a faster machine would touch, because both runs used the same machine. What separates them is entirely the Σα² < ∞ condition Lemma B.3 isolates: a fact about the arithmetic of a sequence of numbers, unmoved by whether that sequence is generated in an afternoon on a PDP-11 or in milliseconds on whatever runs this book. Both reruns therefore answer the book's recurring question the same way twice: neither Sutton's generalization result nor Watkins and Dayan's convergence guarantee was ever waiting on a bigger machine. One is a statement about what a small sample lets a learning rule infer; the other is a statement about what a schedule of numbers is required to sum to. A 2026 datacentre reproduces both in the time it takes to import numpy, and changes the conclusion of neither.
By the early 1990s three lineages of population-based, derivative-free search had matured largely in parallel, each converging on the same basic move: maintain not a single candidate solution but a population of them, evaluate each by a scalar fitness or objective, and generate the next population by some combination of selection and stochastic variation, with no gradient of any error surface ever computed. Xin Yao's 1999 survey gives the family its usual name and its usual membership: "EA's refer to a class of population-based stochastic search algorithms that are developed from ideas and principles of natural evolution. They include evolution strategies (ES), evolutionary programming (EP), and genetic algorithms (GA's)." All three, Yao continues, share "their population-based search strategy," in which "individuals in a population compete and exchange information with each other in order to perform certain tasks." A closely related fourth family, swarm intelligence — particle swarms and ant colonies — arrived slightly later and dispensed even with the genetic vocabulary of parents and offspring, but kept the same population-of-candidates, no-gradient-required core. This chapter's derivations concern the operators these methods actually run: the genetic algorithm's selection, crossover, and mutation on a population of encoded strings; the evolution strategy's self-adapting Gaussian mutation; the particle swarm's velocity update; and, in Section 13.2's closing case, the marriage of genetic search to neural network topology that had defeated a decade of attempts before a change of encoding — not of computer — resolved it.
Vittorio Maniezzo's 1994 paper on evolving neural network topology opens with a textbook statement of what practitioners of the era called the Sequential Genetic Algorithm (SGA), and its five numbered steps are worth reproducing exactly because it is the baseline against which the chapter's other lineages defined themselves — evolution strategies and particle swarm optimization did not modify this cycle so much as dispense with pieces of it one at a time, as the sections below will show. A population X(t) of "individuals," each a string of genes drawn from some alphabet — classically {0,1}, though the allelic value "can potentially be continuous and even structured" — is scored by a fitness function FF computed on each string, in a scheme the paper calls "a computational model inspired by population genetics." The algorithm alternates two kinds of step: a selection step that biases which individuals get to reproduce, and a variation step that perturbs the offspring. Selection, in the standard scheme, is stochastic and proportional to fitness: the paper singles out "roulette wheel selection" as "the most commonly used," a procedure in which "individuals are extracted in probability following a Monte Carlo procedure" with extraction probability "proportional to its fitness" relative to the population average. Crossover is applied with a fixed probability pc: "the individuals of the population are randomly paired," and "each pair is then recombined, choosing one point in accordance with a uniformly distributed probability over the length of the individual strings (parents) and cutting them in two parts, accordingly," so that the offspring are "formed by the juxtaposition of the first part of one parent and the last part of the other parent." Mutation, applied with its own fixed probability pm, "modifies each allele of each individual of the population in probability," typically replacing an allele with a new value drawn uniformly at random. Nothing in this cycle — roulette-wheel selection, single-point crossover, per-allele mutation — refers to a gradient of any kind; the only quantity the algorithm consults is the scalar fitness of each string, evaluated by running the candidate solution and scoring the result, which is exactly why the genetic algorithm could be handed problems where no derivative existed to compute in the first place.
A second lineage, developed independently by Ingo Rechenberg and Hans-Paul Schwefel in Berlin in the 1960s and formalized decades later in Hans-Georg Beyer's treatise "The Theory of Evolution Strategies," dispensed with a discrete alphabet altogether and searched directly in a continuous parameter space. As Nikolaus Hansen's 1996 paper on correlated mutations summarizes it, the evolution strategy "addresses the search problem of minimizing a nonlinear objective function" over a real-valued parameter vector, where "search steps are taken by recombination of already evaluated search points and mutation," and "the mutation is usually carried out by adding a realization of a normally distributed random vector" — that is, given a parent x, an offspring is generated as x' = x + σ·N(0,I), where σ is a step size and N(0,I) a vector of independent standard normal draws. The population notation the field settled on, written (μ,λ) or (μ+λ), records how many parents μ produce how many offspring λ each generation, and whether the parents themselves compete for survival: in a comma strategy, (μ,λ), only the λ offspring are eligible to become the next generation's parents, discarding the current parents outright even if one of them was fitter than every offspring; in a plus strategy, (μ+λ), parents and offspring are pooled and the best μ of the combined set survive. What made the evolution strategy more than hill-climbing with Gaussian noise was that σ itself was placed under evolutionary control rather than fixed by the experimenter: "a dynamic control of certain parameters of the normal mutation distribution is of major importance and is a common feature in evolution strategies. This is often called adaptation or self-adaptation." Beyer's treatise shows why this matters beyond mere convenience: the benefit of recombining multiple parents in a (μ/μ,λ)-ES, he argues, is a "statistical error correction mechanism" he calls "genetic repair," in which averaging several independently mutated parents cancels out the component of each mutation that pointed in a useless direction, leaving the component that improved fitness — a mechanism with no analogue in the single-point crossover of the genetic algorithm, because it operates on continuous vectors by arithmetic averaging rather than on discrete strings by cutting and splicing.
A third lineage abandoned the genetic vocabulary of parents, offspring, and alleles altogether. James Kennedy, a social psychologist, and Russell Eberhart, an electrical engineer, proposed in 1995 an algorithm modeled not on inheritance but on flocking. As Shi and Eberhart's own 1998 follow-up paper describes it, "a population of individuals exists," and instead of using genetic operators, these individuals are evolved "by cooperation and competition among the individuals themselves through generations." Concretely, "each individual is named as a" particle, which "represents a potential solution to a problem," and "each particle is treated as a point in a D-" dimensional space. Every particle i carries a current position Xi and a velocity Vi, written by the paper as "the rate of the position change (velocity) for particle i is represented as Vf(vil,viz, . .. , VD)"; it also remembers the best position it has personally visited, and the "index of the best parhcle among all the particles in the population is represented by the symbol g" — the population's best-so-far. Each generation, a particle's velocity is updated from its previous velocity plus two pulls, one toward its own remembered best and one toward the swarm's remembered best, "where c1 and c2 are two positive constants, rand() and Rand() are two random functions in the" range "[0,1]." One added term is a "cognition" component, representing the private thinking of the particle itself; the other is a "social" component, representing collaboration among the particles. The particle then flies to its new position by simply adding the updated velocity to its old coordinate. No crossover, no encoding of a chromosome, and no alphabet of alleles appear anywhere in the rule — only arithmetic on two vectors per particle, which is why the method was reported to be "robust and fast in solving nonlinear" and non-differentiable problems, at a memory cost of two D-dimensional vectors per particle.
Beyer's treatise gives the evolution strategy's self-adaptation mechanism a name and a chapter of its own: the "(1, λ)-σ-Self-Adaptation," in which the step size σ is no longer set by the experimenter but mutated and inherited exactly as the object parameters are. A parent carries both a position x and a mutation strength σ. To produce an offspring, the strategy first mutates σ itself, before touching x at all, using what the treatise calls the "LOG-NORMAL OPERATOR" (its eq. 7.8): the offspring's strength is obtained from the parent's by an exponential transformation of a normally distributed random number, which in the field's now-standard rendering is written σ' = σ·exp(τ·N(0,1)) — the treatise's own typeset equation did not survive OCR intact, so this is the modernized form of the same rule, not a direct transcription. "The symbol T is called the learning parameter," Beyer writes of the exogenous constant τ that scales the mutation of the mutation strength, and "the correct choice of this parameter influences the adaptation" behavior of the whole strategy: too large a τ and σ thrashes randomly from generation to generation; too small and it barely moves. Only after σ' has been drawn is the object parameter mutated, x' = x + σ'·N(0,I), so that the offspring inherits a self-adapted step size along with a self-adapted position. The mechanism is statistical rather than directed: "an offspring mutation strength" is itself a random draw, and by applying the log-normal rule "this variate is log-normally distributed" — meaning σ can only ever get bigger or smaller by a random multiplicative factor, never by a computed correction toward some target. What makes this random walk on σ useful rather than merely noisy is indirect selection: an offspring that happens to inherit a σ well matched to the local curvature of the fitness landscape is, on average, also an offspring with better x, so selecting on fitness alone — the only quantity the strategy ever measures — implicitly favors the step sizes that produced it, generation after generation, with no separate objective for σ ever written down.
The (1,λ)-σ-self-adaptation of the previous block controls a single scalar step size, the same in every coordinate; it can stretch or shrink the mutation cloud but never reshape it. Nikolaus Hansen's 1996 paper on correlated mutations addresses what happens once that restriction is lifted and the strategy is allowed to adapt not one number but a full covariance matrix. "Concerning the parameters of the normal mutation distribution, which are dynamically controlled, in the most general case the complete covariance matrix is adapted," Hansen writes, and doing so "can be identified with the choice of a linear transformation of" the coordinate system in which the search operates — an ellipsoidal mutation cloud, oriented and stretched to match the local shape of the fitness landscape, rather than the sphere a scalar σ can only ever produce. The scheme the field had converged on by the mid-1990s, denoted CORR-ES, mutates not just step sizes but a set of rotation angles: each offspring's step sizes and angles are formed by taking the arithmetic mean of two parents' corresponding values and then perturbing that mean, so that the covariance matrix itself is inherited and self-adapted exactly as the position vector and the scalar σ were in the simpler strategy. Hansen's paper is concerned with what this buys and what it costs. What it buys is invariance: "Invariance properties of a search algorithm with respect to changes of the" objective function's coordinate system "are extremely attractive," because a strategy that is invariant to a linear change of coordinates behaves identically whether a problem's variables happen to be well-scaled and uncorrelated or badly scaled and entangled — and "therefore the adaptation of the complete covariance matrix should strive for becoming independent from any linear transformation of the object parameters." What it costs is dimensionality: a full covariance matrix on an n-dimensional problem has n(n−1)/2 independent angles in addition to the n step sizes, all of which must themselves be estimated from a population no larger than the ones the scalar strategy used, which is exactly why Hansen's paper is devoted to checking, rather than assuming, that the CORR-ES mutation rule actually achieves the rotational invariance it was designed for.
The original 1995 particle swarm update, as Section 13.1 described, added an unweighted cognition term and an unweighted social term to a particle's previous velocity; nothing in the rule discouraged a particle's velocity from growing without bound as it accumulated pulls toward two different targets generation after generation, which is why the method needed an externally clamped maximum velocity to keep particles from flying off the search space entirely. Shi and Eberhart's 1998 paper diagnoses this as a missing balance between global and local search and repairs it with a single multiplicative term: "a inertia weight w is brought into the equation (1) as shown in equation (2). This w plays the role of balancing the global search and local search." In the field's now-standard notation — the paper's own typeset equations did not survive OCR cleanly, so this is a modernized transcription of the same rule, not a direct quotation of it — the update for the d-th coordinate of particle i becomes v_id(t+1) = w·v_id(t) + c1·rand()·(p_id − x_id(t)) + c2·Rand()·(p_gd − x_id(t)), x_id(t+1) = x_id(t) + v_id(t+1), where p_id is the particle's own best-remembered position, p_gd is the swarm's best-remembered position, c1 and c2 are fixed constants weighting the cognition and social pulls, and rand() and Rand() are independent draws from [0,1]. Sweeping w across fixed values from 0.1 to 2.0 while holding everything else constant, the paper reports that "the bigger the inertia weight w is. the less dependent on initial population the solution is. and the more capable to exploit new areas the PSO is" — a large w favors broad exploration at the cost of precision, a small w favors local refinement at the cost of getting stuck — and concludes that a band of moderate values, roughly 0.9 to 1.2, gave the best overall balance between reliability and speed of convergence in these trials: "It is concluded that the PSO with the inertia weight in the range" of that band, taken "on average will have a better performance" than values well outside it. Rather than commit to one fixed value, the paper's second proposal lets w itself change over the course of a run: "we defined the inertia weight w as a decreasing function of time instead of a fixed constant. It started with a large value 1.4 and linearly decreased to 0 when the iteration number reached 4000." The schedule is meant to grant broad exploration early, when the swarm knows little about the landscape, and narrow, exploitative search later, once good regions have been located — the same explore-then-exploit shape as a simulated-annealing temperature schedule, arrived at independently and folded into a single scalar multiplying every particle's inherited velocity.
Maniezzo's 1994 genetic algorithm for network topology, discussed in Section 13.1, ran into a problem that no amount of computation could fix by itself: two networks that compute the same function can be encoded as wildly different genomes, so that crossing over two fit parents by cutting and splicing their strings, in the ordinary single-point manner, produces an offspring with damaged, misaligned structure more often than not. Kenneth Stanley and Risto Miikkulainen's NEAT algorithm solves this "competing conventions" problem not with more search but with better bookkeeping: every gene added anywhere in the population is stamped with a permanent serial number at the moment of its invention. "Whenever a new gene appears (through structural mutation), a global innovation number is incremented and assigned to that gene," and crucially, "in the future, whenever these genomes mate, the offspring will inherit the same innovation numbers on each gene; innovation numbers are never changed." Because two genomes that both trace back to the same historical mutation carry the same innovation number on the corresponding gene no matter how much unrelated structure has grown up around it elsewhere, crossover no longer needs to guess which parts of two genomes correspond: "the genes in both genomes with the same innovation numbers are lined up. These genes are called matching genes," while genes with no counterpart are disjoint or excess; in composing the offspring, genes are chosen at random from either parent at each matching locus, while every excess or disjoint gene is inherited from whichever parent is the fitter of the two. The same innovation numbers give NEAT a natural distance metric between two genomes, a weighted sum of how much of their history they do not share: δ = c1·E/N + c2·D/N + c3·W̄ (eq. 1), where E and D count excess and disjoint genes, W̄ is the average weight difference on genes that do match, N normalizes for genome size, and "the coefficients c1, c2, and c3 allow us to adjust the importance of the three factors." This distance is put to work immediately, protecting structural innovation from being outcompeted before it has had a chance to prove itself: genomes are grouped into species by δ against a threshold δt, and within a species fitness is shared rather than compared directly against the whole population, f'_i = f_i / Σ_j sh(δ(i,j)) (eq. 2), where "the sharing function sh is set to 0 when distance δ(i,j) is above the threshold δt; otherwise, sh(δ(i,j)) is set to 1," so that the sum in the denominator simply counts how many other individuals share a species with organism i. A large, newly speciated topology is thus compared for fitness only against its own small niche rather than against an entire population of already-optimized smaller networks, giving a structural mutation time to be refined by later generations before it is culled — the encoding-level fix, arrived at without any change to the raw compute available, that let genetic search finally out-compete hand-designed topology and, in the chapter's closing case in Section 13.3, hand-tuned backpropagation as well.
Three claims from this chapter's derivations are checkable on a laptop in the time it takes to import numpy: that the (1,λ)-σ-self-adapted evolution strategy converges geometrically on a simple landscape without any experimenter-set schedule for its step size; that an inertia-weighted particle swarm behaves the way Shi and Eberhart's sweep reported, doing better with moderate w and better still with a decreasing schedule than with any single fixed value; and that a bare textbook genetic algorithm, of the kind Section 13.1 quoted from Maniezzo, actually optimizes the continuous weights of a small neural network better than blind random search at an equal computational budget. None of these experiments strains a workstation from any decade — a population of ten to forty candidates run for a few hundred generations on a two- or ten-dimensional problem was never the part of this chapter's methods that was hardware-bound. The reruns below use modern hardware (numpy on whatever machine is running this book) in place of whatever workstation the original authors used, and the sphere-model, Schaffer's f6, and XOR problems in place of the exact test functions each original paper may have favored, since the point of a rerun in this chapter is not to reproduce a specific 1990s benchmark number but to ask, of each qualitative claim, whether it depends on the arithmetic getting done fast, or on something else.
The first rerun implements the (1,λ)-σ-self-adaptation rule of Section 13.2's first block directly: a single parent (x, σ) produces λ = 10 offspring each generation by first drawing σ' = σ·exp(τ·N(0,1)) with τ = 1/√(2N), then x' = x + σ'·N(0,I), on the N = 10-dimensional sphere function f(x) = Σxᵢ², minimized from a random start with σ initialized to 1. Averaged over five seeds (run 272-es-selfadapt-sphere.py), the best offspring's sphere value falls from 12.29 at generation 0 to 0.0088 at generation 50, to 3.6×10⁻⁵ at generation 100, to 2.3×10⁻⁸ at generation 150, to 1.3×10⁻¹⁰ by generation 199 — fourteen orders of magnitude of improvement with no gradient of the sphere function ever computed, only a scalar fitness compared between ten offspring each generation. Fitting a line to log(fitness) against generation over the second half of the run, generations 50 through 199, gives a slope of −0.132: the value is falling by a roughly constant multiplicative factor every generation, the geometric convergence Beyer's treatise characterizes as "linear convergence order" on the sphere model — linear meaning linear in the log of the fitness, a constant fractional improvement per step, not a constant absolute one. This is exactly what self-adaptation is supposed to deliver without anyone hand-tuning a cooling schedule for σ: nothing in the rerun's code ever reads generation number to decide how far to step; σ is left entirely to the log-normal mutation-and-selection process derived in Section 13.2, and it finds, unassisted, a step size that keeps shrinking at the rate the sphere model's curvature calls for.
The second rerun implements the inertia-weighted update of Section 13.2's third block on Schaffer's f6 function, f(x,y) = 0.5 + (sin²√(x²+y²) − 0.5)/(1 + 0.001(x²+y²))², whose global minimum of 0 sits at the origin surrounded by concentric rings of shallow local minima that have made f6 a standard PSO stress test since the inertia-weight paper itself used it. A swarm of 20 particles, velocity-clamped to ±2.0 and initialized uniformly on [−100,100]², is run for up to 4000 iterations with cognition and social weights fixed at c1 = c2 = 2.0, the values standard across this literature, and a run counts as a success if the best-known value falls below 10⁻⁴ (run 273-pso-schaffer-f6.py, 30 seeds per condition). Sweeping the fixed inertia weight w reproduces the shape the original paper reports: very small w gets stuck early, converging fast but to the wrong point — w = 0.1 succeeds on only 18 of 30 seeds, w = 0.5 on 12 of 30, in a mean 206–217 iterations on the runs that do succeed — while very large w wanders too freely to settle, w = 2.0 succeeding on only 11 of 30 seeds and needing the full 4000 iterations even then. In between, success climbs and holds: w = 0.9 succeeds on 24 of 30 seeds, averaging 1557 iterations, and w = 1.0 and w = 1.1 — both inside the "roughly 0.9 to 1.2" band Section 13.2 quoted as the paper's own best fixed setting — succeed on all 30 seeds, though at real cost in iterations, averaging 3231 and 3846 respectively; w = 1.2 slips to 25 of 30 and w = 1.4 to 23 of 30, each averaging upward of 3600 iterations. The rerun does not find a single fixed value inside that band that is both fast and reliable — reliability is highest near its center while mean iterations-to-success keeps rising as w increases through it — but it reproduces the paper's central qualitative claim, that reliability rises through a middle band of w before falling away on both sides of it. The decreasing schedule beats every fixed value on both counts at once rather than trading one for the other: run with w falling linearly from 1.4 to 0 over the 4000 iterations, exactly as the paper specifies, it succeeds on all 30 seeds — matching the most reliable fixed settings — while doing so in a mean of 1584 iterations, under half what w = 1.0 or w = 1.1 needed to reach the same reliability. That is the qualitative claim Section 13.2 drew from the paper's second proposal: a schedule that starts broad and narrows over the run is not a mere compromise between exploration and exploitation but can dominate every single constant in the sweep, matching the best fixed value's reliability while converging several times faster — a result no fixed w can produce, since no one number can be both the best explorer at iteration ten and the best exploiter at iteration three thousand.
The third rerun tests the claim that took up the second half of Section 13.1: that Maniezzo's Sequential Genetic Algorithm — roulette-wheel selection proportional to fitness, single-point crossover on a fixed-length string, and per-allele mutation, with no gradient of any kind consulted — actually finds better weights for a small neural network than blind random search does, at the same number of network evaluations. The test network is a 2-2-1 feedforward net with bias on both layers (nine real-valued weights and biases in all) trained on XOR, the four-pattern problem whose non-linear separability is this chapter's standard stand-in for "a function no single-layer perceptron can represent," scored by mean squared error between the network's sigmoid output and the target on all four patterns. Each string in the genetic algorithm's population is simply the nine weights concatenated, with no further encoding: fitness is the reciprocal of mean squared error (so lower error means a higher chance of being drawn by the roulette wheel), a single elite individual is copied unchanged into the next generation, and every other offspring is built by single-point crossover between two parents drawn proportional to fitness followed by an independent chance, per allele, of adding Gaussian noise to that one weight — the SGA cycle of Section 13.1's block 2, transcribed onto continuous alleles rather than a binary alphabet, exactly as Maniezzo's own paper allows ("the allelic value ... can potentially be continuous"). A population of 30 runs for 200 generations, so each of the 30 independent seeds costs 30 × 201 = 6,030 evaluations of the network, counting the final generation's evaluation; the random-search baseline for the same seed is simply 6,030 independent draws of nine weights, uniform on [−2,2], keeping the best mean squared error seen (run 275-ga-vs-random-xor.py).
At that equal budget of 6,030 evaluations per seed, the genetic algorithm wins decisively and on every seed: over 30 independent runs its final mean squared error averages 0.0657 with standard deviation 0.0663, against a random-search baseline of 0.2256 mean, 0.0046 standard deviation, at the identical evaluation count (run 275-ga-vs-random-xor.py). The comparison is not close and not lucky — the genetic algorithm's best final error beats the random search's on all 30 of 30 seeds — but the two distributions tell different stories about why. Random search is metronomic: with 6,030 independent uniform draws over a nine-dimensional box, it reliably lands somewhere in the shallow basin around 0.22–0.23 mean squared error and essentially never does better or worse than that, since blind sampling has no way to concentrate its draws near a promising point once it has found one. The genetic algorithm's own spread is wider than the baseline's — some seeds land it at 0.0002 or 0.0006, a near-perfect XOR fit, while a recognizable cluster of other seeds stalls around 0.126, a local optimum where two of the four XOR patterns are fit and two are not — but even its worst seeds usually beat random search's best, because selection and crossover let a lineage that has found a good weight combine it with another lineage's improvement rather than discarding all memory of the search every single draw. This is Maniezzo's claim from Section 13.1 holding up exactly where the chapter said to check it: nothing about roulette-wheel selection, single-point crossover, or per-allele mutation requires a faster computer to work — the mechanism that lets the genetic algorithm beat random search at equal cost is that it remembers which strings scored well and recombines them, a bookkeeping advantage available to a population of thirty on any machine from any decade, not a consequence of how many evaluations that machine can run per second.
Most of the chapters so far in this volume have assumed a teacher: a label, a reward, a target value the learner is scored against — perceptron, backpropagation, the support vector machine, boosting, temporal-difference learning. Two chapters already broke that assumption in part: chapter 4's energy-based networks, where the Hopfield model retrieves a stored pattern from a noisy cue with no target to match and the Boltzmann machine is trained to reproduce a data distribution rather than a mapping, and chapter 9's treatment of EM and mixture models, built for exactly the case where the population an observation came from was never recorded. This chapter makes that label-free setting the rule rather than the exception. The data arrive as a set of vectors x1, . . . , xM ∈ R^N with no y attached, and the task is to say something true about their structure anyway — to find the groups they fall into, or the low-dimensional surface they actually live on, or both. This is the oldest problem in statistics wearing a newer name; what changes across the period covered here is not the goal but the geometry brought to bear on it. Classical technique treated the data as living in, or near, a linear subspace of the ambient space and asked for the best-fitting subspace (principal component analysis) or the configuration of points that best preserves pairwise distances (multidimensional scaling). The nonlinear techniques that are this chapter's destination — Isomap, locally linear embedding, Laplacian eigenmaps, spectral clustering — keep the same eigenvector machinery but abandon the linearity assumption, replacing a single global subspace with a manifold: a surface that looks locally like a plane but is globally curved, so that no linear projection can unroll it without distortion.
The nineteenth-century baseline is Karl Pearson's and Harold Hotelling's principal component analysis, restated by Schölkopf, Smola, and Müller in 1998 in the language this chapter will need: "Principal component analysis (PCA) is a powerful technique for extracting structure from possibly high-dimensional data sets." Concretely, PCA diagonalizes the covariance matrix of the centered data, C = (1/M) Σ_j x_j x_j^T, and re-expresses each observation in the eigenbasis of C. "PCA is an orthogonal transformation of the coordinate system in which we describe our data. The new coordinate values by which we represent the data are called principal components," and when a handful of the largest eigenvalues account for most of the variance, those few coordinates serve as a compressed, decorrelated description of the whole data set. Schölkopf and colleagues' own contribution — kernelizing this eigenvalue problem so that the covariance is computed in a nonlinearly mapped feature space rather than the input space — is itself billed as "an example of applying this method in the domain of unsupervised learning, to obtain a nonlinear form of PCA": the same eigen-decomposition machinery, pointed at a feature space rich enough to make curved structure look linear. It is a first move away from strict linearity, but the mapping is fixed in advance by the choice of kernel rather than read off the data, and it says nothing about neighborhoods or geodesics. The manifold methods below make the opposite bet: keep the input space Euclidean, and let the data's own local neighborhood structure dictate the embedding.
Tenenbaum, de Silva, and Langford open their 2000 paper on Isomap with the general version of the problem: "Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations." Their diagnosis of where PCA and MDS fall short is specific: those methods are "guaranteed to discover the true structure of data lying on or near a linear subspace," but real data rarely obliges. Their running example is a face database — 64×64 pixel images, so a nominal 4096 input dimensions — photographed under varying pose and lighting; despite the vast ambient dimension, "all of the images lie on an intrinsically three-dimensional manifold, or constraint surface, that can be parameterized by two pose variables plus an azimuthal lighting angle." Their algorithm is offered as a repair: "Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions." This is the manifold hypothesis in its founding form: the data's few true degrees of freedom (pose angles, lighting angle) are related to the pixel values by a smooth but nonlinear map, and the job of dimensionality reduction is to invert that map — to recover the pose coordinates from the pixels — without ever being told what the map is.
The other half of unsupervised learning asks a discrete rather than a continuous question. Von Luxburg's tutorial states it plainly: "Clustering is one of the most widely used techniques for exploratory data analysis, with applications ranging from statistics, computer science, biology to social sciences," and the goal, stripped of any particular algorithm, is that "the intuitive goal of clustering is to divide the data points into several groups such that points in the same group are similar and points in different groups are dissimilar to each other." For decades this looked like a separate problem from dimensionality reduction, solved by separate tools — k-means's alternating reassignment of points to the nearest of k centroids, hierarchical agglomeration, mixture models fit by the EM algorithm of chapter 9. What unifies this chapter is the discovery, made independently by several groups around 2000–2003, that clustering and manifold embedding are the same computation performed on the same object: a graph built from pairwise similarities, and its Laplacian. Belkin and Niyogi, introducing Laplacian eigenmaps, make the connection explicit and note its limits in the same breath: the locality-preserving embedding "implicitly emphasizes the natural clusters in the data," so that "dimensionality reduction and clustering are two sides of the same coin," while cautioning that "not all data sets necessarily have meaningful clusters," in which case a global method such as PCA or Isomap is the more honest tool. Section 2 derives the four methods this setup has named — kernel PCA, Isomap, locally linear embedding, and Laplacian eigenmaps/spectral clustering — as four different answers to the eigenvalue problem that follows once a notion of local similarity has been fixed.
Chapter 7 already derived the first of this setup's four answers in full: Schölkopf, Smola, and Müller's kernel PCA replaces the covariance eigenproblem λv = Cv of ordinary PCA with the kernel-matrix eigenproblem Mλα = Kα, where K_ij = (Φ(x_i)·Φ(x_j)) is built from a chosen kernel k(x,y) = (Φ(x)·Φ(y)) rather than read off the data's own neighborhoods. That derivation turned on one fact worth restating here because every method below reuses it: since the feature-space covariance C̄ satisfies C̄V = (1/M)Σ_j Φ(x_j)(Φ(x_j)·V), any eigenvector V with nonzero eigenvalue must lie in the span of the mapped data, so that, in the paper's words, "there exist coefficients α_i (i = 1, . . . , M) such that" V = Σ_{i=1}^M α_i Φ(x_i) — an M-dimensional problem in the sample size, never the ambient or feature dimension, however large the latter may be. Kernel PCA fixes the geometry in advance, in the choice of kernel; what the mapping does to neighborhoods is implicit in k and never inspected. The three methods this section adds — Isomap, locally linear embedding, and Laplacian eigenmaps — invert that choice. Each first builds an explicit neighborhood graph from the Euclidean distances between the N-dimensional data points themselves, and only then poses an eigenvalue problem on a matrix derived from that graph: the geodesic-distance matrix for Isomap, the reconstruction-weight matrix for LLE, the graph Laplacian for Belkin and Niyogi. The unknown is no longer a feature-space direction but a set of low-dimensional coordinates y_1, . . . , y_M, one per data point, chosen so that some local relationship measured in the input space — geodesic distance, linear reconstruction, the weighted difference between neighbors — is preserved as closely as possible among the y_i. In every case the "closely as possible" is cashed out as a quadratic form in y, and the constrained minimizer of a quadratic form is again an eigenvector problem: the machinery does not change from kernel PCA to Isomap to Laplacian eigenmaps, only the matrix that machinery is applied to, and what that matrix is built to preserve.
Tenenbaum, de Silva, and Langford's Isomap keeps the second stage of classical multidimensional scaling — an eigendecomposition that turns a matrix of squared distances into coordinates — and replaces only the distances fed into it. "The complete isometric feature mapping, or Isomap, algorithm has three steps." Step one turns the raw data into a graph: it "determines which points are neighbors on the manifold M, based on the distances dX(i,j) between pairs of points i,j in the input space X," connecting each point either to all points within a fixed radius ε or to its K nearest neighbors, with "edges of weight dX(i,j) between neighboring points." Step two approximates the geodesic — the true, manifold-respecting distance — for pairs of points that are not neighbors: Isomap "estimates the geodesic distances dM(i,j) between all pairs of points on the manifold M by computing their shortest path distances dG(i,j) in the graph G," summing short Euclidean hops along the graph rather than measuring the direct, potentially manifold-cutting straight line. This is the paper's diagnosis of why PCA and classical MDS fail on curved data made operational: on a two-dimensional "Swiss roll," points far apart on the underlying manifold as measured by geodesic distance can appear deceptively close in the high-dimensional input space as measured by straight-line Euclidean distance, so only graph shortest-paths — not ambient Euclidean distance — track the surface's true geometry. Step three hands the resulting matrix of estimated geodesic distances D_G to classical MDS unmodified: an embedding Y is sought that minimizes E = ‖τ(D_G) − τ(D_Y)‖_{L²}, where D_Y is the matrix of Euclidean distances in the embedding and τ is the standard MDS operator, which "converts distances to inner products," turning a distance matrix into a Gram matrix whose eigendecomposition gives the embedding directly: "the global minimum of Eq. 1 is achieved by setting the coordinates yi to the top d eigenvectors of the matrix t(DG)" (the paper's τ rendered as "t" in the source text). Applied to N = 698 images of a face rendered under varying pose and lighting, this recovered a three-coordinate embedding in which "Each coordinate axis of the embedding correlates highly with one degree of freedom underlying the original data: left-right pose (x axis, R 5 0.99), up-down pose ( y axis, R 5 0.90), and lighting direction (slider position, R 5 0.92)" — pose and lighting angles recovered from pixels alone, with no model of either ever supplied to the algorithm. The only new ingredient relative to classical MDS is graph shortest-path distance in place of ambient Euclidean distance; the eigendecomposition that turns distances into coordinates is unchanged.
Roweis and Saul's locally linear embedding poses no distance-preservation problem at all, global or local; it poses two separate least-squares problems, both linear, chained together. The corpus copy of their Science report preserves the paper's method notes and figure captions rather than its opening page, so the derivation below is reconstructed from those notes, in the paper's own symbols; the surrounding exposition is not available to quote directly. Step one, in the paper's summary of its own figure, is to "assign neighbors to each data" point — by K nearest neighbors, as elsewhere in this chapter — and then to compute the weights that best "reconstruct" each point from those neighbors, "solving the constrained least-squares problem in Eq. 1." The constraint is that the weights from any one point sum to one, so the reconstruction is invariant to rotation, rescaling, and translation of the neighborhood: for a data point x with neighbors η_j and sum-to-one weights w_j, the reconstruction error ‖x − Σ_j w_j η_j‖² "is minimized in three steps" — form the neighborhood correlation matrix C_jk = η_j·η_k, invert it, and solve for the w_j in closed form using a Lagrange multiplier that enforces the sum-to-one constraint. This is a per-point linear-algebra problem, with no eigenvectors and no iteration: every row of the weight matrix W is computed independently of every other. Step two fixes those weights and asks for low-dimensional vectors y_i that the same weights reconstruct: the embedding vectors are "found by minimizing the cost function" Φ(Y) = Σ_i‖Y_i − Σ_j W_ij Y_j‖² over Y_i with fixed weights W_ij, subject to the embedding being centered at the origin and having unit covariance, which rules out the trivial all-zero solution and an arbitrary overall scale. Expanded, this cost is a quadratic form Φ(Y) = Σ_ij M_ij(Y_i·Y_j) in the symmetric N×N matrix M_ij = δ_ij − W_ij − W_ji + Σ_k W_ki W_kj, which the paper notes "can be stored and manipulated as the sparse matrix (I − W)ᵀ(I − W)"; the optimal embedding "is found by computing the bottom d 1 1 eigenvectors of this matrix," discarding the very bottom one, since "the bottom eigenvector of this matrix, which we discard, is the unit vector with all equal components" and represents a free translation mode of eigenvalue zero. The remaining d eigenvectors, associated with the smallest nonzero eigenvalues of M, are the embedding coordinates. Isomap solves one global eigenproblem on a dense matrix of estimated geodesic distances between every pair of points; LLE solves N small, independent least-squares problems to build a sparse matrix, then one eigenproblem at the bottom of that matrix's spectrum rather than the top of a Gram matrix's — reconstruction error, not distance or variance, is the quantity being preserved.
Belkin and Niyogi's Laplacian eigenmaps takes the neighborhood graph one step further than either Isomap or LLE: rather than approximating distances along the graph or reconstructing each point from its neighbors, it works directly with the graph's Laplacian matrix, the object spectral graph theory had already studied for clustering and partitioning before anyone applied it to embedding. The graph is built the same way as in the other two methods — "we construct a weighted graph with k nodes, one for each point, and a set of edges connecting neighboring points" — with an edge whenever two points are within an ε-ball or among each other's n nearest neighbors, and each edge weighted either by the heat kernel Wij = e^(−‖xi−xj‖²/t) or, in the simplest case, by "Wij = 1 if vertices i and j are connected by an edge and Wij = 0 if vertices i and j are not connected by an edge." From the weight matrix W and its diagonal degree matrix D (Dii = Σj Wji), "L = D −W is the Laplacian matrix." The embedding problem is posed exactly as the setup in block 1 anticipated: find y = (y1, . . . , yn)ᵀ, one real number per data point, so that connected points stay as close together as possible, which the paper cashes out as a criterion "to minimize the following objective" — the sum Σij(yi − yj)²Wij — "under appropriate constraints." That sum is a quadratic form in disguise: expanding it using the symmetry of W and the definition of D shows Σi,j(yi−yj)²Wij = 2yᵀLy, and, in the paper's own aside, "this calculation also shows that L is positive semidefinite," since a sum of squares can never go negative. Minimizing yᵀLy under the scale-fixing constraint yᵀDy = 1 is a generalized Rayleigh-quotient problem, and its constant solution y = 1 (which collapses every point onto the same real number) has to be excluded by an orthogonality constraint, exactly as the trivial all-ones eigenvector had to be discarded in LLE's cost function. The vector "y that minimizes the objective function is given by the minimum eigenvalue solution to the generalized eigenvalue problem" Ly = λDy, taking the eigenvector belonging to the smallest nonzero eigenvalue rather than the zero eigenvalue belonging to the constant vector; embedding into m dimensions instead of one takes the next m eigenvectors of the same problem in order of increasing eigenvalue. The heat-kernel weight is not an arbitrary choice of similarity function: the paper derives it from the manifold's Laplace-Beltrami operator, whose eigenfunctions are the smooth, continuous analogue of what the graph Laplacian's eigenvectors approximate on a finite sample, and shows that "in order to ensure that the approximation matrix is positive semidefinite, we compute the graph Laplacian with the following weights" — the same Gaussian that reappears, unannounced, as the kernel of choice throughout the rest of the chapter. Isomap preserves geodesic distance, LLE preserves reconstruction weight, and Laplacian eigenmaps preserves this one local, pairwise quantity — the weighted squared difference between neighbors — but the underlying move is now visibly the same move three times: state the preservation criterion as a quadratic form in the unknown coordinates, fix the trivial solution by a normalization constraint, and read the answer off the extreme eigenvectors of the resulting matrix.
Belkin and Niyogi's own remark that dimensionality reduction and clustering are "two sides of the same coin" (quoted in this chapter's setup) is not a metaphor; von Luxburg's tutorial on spectral clustering shows the two problems solved by the identical generalized eigenvalue problem Ly = λDy, aimed at a different objective. Clustering starts from a graph-cut question rather than a distance- or reconstruction-preservation question: "given a similarity graph with adjacency matrix W, the simplest and most direct way to construct a partition of the graph is to solve the mincut problem," cutting the graph into pieces so as to minimize the total edge weight severed. Unweighted mincut is easy but nearly useless, since its unconstrained optimum typically isolates a single outlying vertex rather than producing balanced groups; forcing the pieces to be comparably sized — the RatioCut and Ncut objectives — turns the discrete search over graph partitions into a combinatorial problem with no efficient exact solution. "Spectral clustering is a way to solve relaxed versions of those problems": drop the requirement that the indicator vector f (which records which side of the cut each vertex falls on) take only two discrete values, and allow it to range over all of R^n instead, and the discrete optimization becomes a continuous one — minimize f'Lf subject to f being orthogonal to the constant vector and normalized in length — which is exactly the Rayleigh-quotient problem Laplacian eigenmaps already solves for a different purpose. "So we can approximate a minimizer of RatioCut by the second eigenvector of L," the eigenvector belonging to the smallest nonzero eigenvalue, precisely the vector Belkin and Niyogi's block above uses to embed rather than to cut. The unnormalized graph Laplacian that both problems share obeys three properties von Luxburg proves from the definition L = D − W alone: "L is symmetric and positive semi-definite," "the smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1," and the number of zero eigenvalues counts the graph's connected components — the same positive-semidefiniteness and the same trivial constant eigenvector that Belkin and Niyogi's derivation excludes by an orthogonality constraint reappears here as the mathematical fact that makes a naturally disconnected data set trivially separable by eigenvectors alone, no optimization needed. What turns eigenvectors into a partition is one further step absent from embedding: cluster the eigenvector coordinates themselves. In the unnormalized algorithm, having computed "the first k eigenvectors u1,...,uk of L," one forms the matrix U with those eigenvectors as columns and treats each row of U as a point in R^k, then clusters the rows of U — the points (yi), i = 1, . . . , n, in R^k — with the k-means algorithm into clusters C1, . . . , Ck, running the oldest unsupervised algorithm in this chapter's toolkit, not on the original data, but on coordinates the Laplacian's spectrum has already produced. Shi and Malik's normalized version replaces L by the generalized eigenproblem Lu = λDu — the identical equation Belkin and Niyogi solve — while Ng, Jordan, and Weiss's version instead eigendecomposes the symmetrically normalized Laplacian L_sym and rescales each row of U to unit length before the k-means step, a normalization von Luxburg shows is necessary precisely when the graph's vertex degrees are uneven, since Rayleigh-Ritz guarantees "its solution g is given by the second eigenvector of Lsym" (the paper's subscript notation for the symmetric normalization, rendered inline here) only once that rescaling has been undone. Three papers, three slightly different Laplacians and slightly different objectives — cut a graph, preserve a local distance, reconstruct from neighbors — arrive at the same generalized eigenvalue problem because all three are Rayleigh quotients in disguise, and the Rayleigh quotient's minimizer, constrained to be orthogonal to whatever mode is already known to be trivial, is always an eigenvector.
Section 2's four derivations each came with a founding empirical claim: kernel PCA recovers nonlinear structure invisible to linear PCA; Isomap recovers the true pose and lighting parameters behind a face image from pixels alone; locally linear embedding does the same with a different, purely local cost function; Laplacian eigenmaps and spectral clustering solve the identical eigenproblem for the discrete question of group membership. None of Tenenbaum, de Silva, and Langford's 698 face images, Roweis and Saul's 2,000 face and word vectors, or Belkin and Niyogi's original test sets are bundled with this book's computing environment, so this section reruns the qualitative claims on the classic synthetic surrogate that stood in for that face manifold from the start of the literature: the "Swiss roll," the two-dimensional sheet curled into a spiral in three-dimensional space that Tenenbaum and colleagues themselves used, alongside the face data, to make the case against PCA and MDS visually — "for the Swiss roll, where classical methods fail, the residual variance of Isomap correctly bottoms out at d 5 2" (the source's OCR renders the equals sign as "5") — and a second synthetic set, two interleaved half-moons, standing in for Belkin and Niyogi's and von Luxburg's non-convex cluster examples. Every number below comes from a scikit-learn implementation of Isomap, locally linear embedding, and spectral clustering run on data generated in this session, not from the papers' own reported figures; where the rerun disagrees with or sharpens the original claim, that disagreement is reported rather than smoothed over.
The rerun generated 1,500 points on a noisy Swiss roll (scikit-learn's make_swiss_roll, noise 0.05), recorded the two true generating coordinates — position unrolled along the spiral, and height along the roll's axis — and then asked ordinary PCA and an Isomap built exactly as section 2 described, "based on the distances dX(i,j) between pairs of points i,j in the input space X... to all of its K nearest neighbors" (K = 12 here, in place of the paper's unspecified per-experiment value), to recover two coordinates from the raw three-dimensional point cloud alone, with no access to the generating parameters. Correlating each method's best output coordinate against the true unrolled position gave |r| = 0.224 for PCA's top two components against |r| = 0.992 for Isomap; against the roll's height coordinate, PCA scored |r| = 0.167 against Isomap's |r| = 0.997. This is the Science paper's own diagnostic, run on a different dataset than the face photographs but the identical failure mode: "Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations." Where Tenenbaum, de Silva, and Langford reported correlations of R = 0.99, 0.90, and 0.92 between Isomap's three coordinates and the face manifold's pose and lighting angles, the rerun's correlations of 0.992 and 0.997 on the two-dimensional Swiss roll are, if anything, cleaner — consistent with the Swiss roll being a synthetic surface with an exactly known, noise-free intrinsic geometry rather than a photograph subject to real optical and pose-sampling irregularities.
Locally linear embedding, run on the identical 1,500-point Swiss roll with the same neighborhood size (K = 12, matching Roweis and Saul's own practice of fixing K as the method's one free parameter), recovered the unrolled position coordinate just as well as Isomap: |r| = 0.992 against the true position, against Isomap's 0.992 — indistinguishable to three figures. Against the roll's height coordinate, however, the rerun found a real difference the two papers' own comparisons only gesture at: LLE's best correlation was |r| = 0.728, well short of Isomap's 0.997. The gap is consistent with what the two derivations in section 2 predict. Isomap solves one global eigenproblem on a matrix of estimated geodesic distances between every pair of points, so a single global embedding criterion is optimized against the whole manifold at once; LLE reconstructs each point only from its immediate neighbors and never checks whether two well-separated patches of the manifold sit at consistent relative scale or orientation once assembled, so a direction that is locally well constrained by nearby points (position along the spiral, where curvature is high and neighbors are informative about direction) comes through more reliably than one where the roll's local neighborhoods are nearly flat and provide less leverage on the global coordinate (height). Roweis and Saul's own comparison of the two methods says as much without needing the rerun to confirm it: "Isomap's embeddings, however, are optimized to preserve geodesic distances between general pairs of data points, which can only be estimated by computing shortest paths through large sublattices of data. LLE takes a different approach, analyzing local symmetries, linear coefficients, and reconstruction errors instead of global constraints, pairwise distances, and stress functions." The rerun's asymmetric correlations are one concrete instance of that stated tradeoff between a global and a purely local criterion, not a contradiction of either paper's claims.
The clustering half of section 2's argument — that Laplacian eigenmaps and mincut-relaxation spectral clustering solve the same generalized eigenproblem Ly = λDy — makes an empirical prediction distinct from anything Isomap or LLE claim: that clustering directly on graph-Laplacian eigenvectors should succeed on data where ordinary k-means, minimizing squared Euclidean distance to a centroid, cannot. Von Luxburg's tutorial states the intended contrast without yet giving numbers: the spectral representation "enhances the cluster-properties in the data, so that clusters can be trivially detected in the new representation," at which point, the tutorial goes on, plain k-means run on that new representation encounters no difficulty at all — the claim being about the new, eigenvector representation, not the original coordinates. The rerun tested exactly this on two interleaved half-moons (600 points, scikit-learn's make_moons, noise 0.06) — a textbook non-convex pair of clusters no partition into two centroid-based Euclidean regions can separate correctly. Plain k-means on the raw two-dimensional coordinates scored an adjusted Rand index of 0.255 against the true half-moon labels (1.0 is perfect agreement, 0.0 is chance); spectral clustering — a k-nearest-neighbor similarity graph, its Laplacian, and k-means applied only to the resulting eigenvectors, exactly the pipeline the tutorial describes — scored 1.0, perfect recovery. The qualitative claim is not merely confirmed but sharpened to the point of triviality on this synthetic case: k-means fails not partially but almost at the level of a random split, while the identical clustering rule applied after the Laplacian eigenvector step recovers the two moons exactly.
Isomap's second step — turning a sparse neighborhood graph into a dense matrix of all-pairs shortest-path distances — is the one place in this chapter's four derivations where the original paper names its own computational cost directly, in a footnote on its own shortest-path procedure: "This procedure, known as Floyd's algorithm, requires O(N3) operations," the note goes on to add that more efficient algorithms exploiting the sparse structure of the neighborhood graph exist elsewhere in the literature, without the paper itself using them. This is worth pausing on against the book's recurring question, because it is not a single verdict but two different costs bundled under one name. Timed on the rerun's own Swiss roll, scikit-learn's Isomap — which does not use Floyd's algorithm at all, instead running Dijkstra's algorithm from every node against the graph's sparse adjacency structure, exactly the substitution the original paper gestured at but did not implement — completed in 0.10 s at N = 698 (the size of the original face experiment), 0.43 s at N = 1,500, and 1.92 s at N = 3,000, on ordinary CPU hardware and without any GPU involved. The wall-clock ceiling the 2000 paper danced around is gone: at the scale of its own face experiment, and at several times that scale, the shortest-path step is now instantaneous by any standard the paper's authors would have recognized. But the underlying quantity has not become smaller. The dense N×N matrix of pairwise geodesic distances that step two must produce, whatever algorithm fills it in, still occupies N² floating-point numbers and cannot be built for a corpus of the scale later chapters take for granted — a million points would demand a trillion-entry matrix, on the order of eight terabytes at double precision, an amount no single machine holds in memory whether the year is 2000 or 2026. This is the distinction the book has drawn before: the specific bottleneck the original authors named — an O(N³) shortest-path routine on a 1990s workstation — is gone, dissolved by better algorithms on faster hardware. The bottleneck the derivation itself imposes — an eigendecomposition that needs every pairwise distance at once, and so a matrix quadratic in the number of points — is mathematics, not machine speed, and it still binds Isomap to samples of thousands rather than millions of points regardless of what datacentre stands behind it.
Taken together, the four reruns confirm section 2's central claim rather than complicating it: every method's numerical output tracks its own stated preservation criterion, and the criteria are genuinely different from each other even though the closing eigenvector step is shared. Isomap's global geodesic-distance criterion recovered both Swiss-roll coordinates at correlation above 0.99; LLE's purely local reconstruction criterion matched Isomap on the coordinate its neighborhoods could see clearly (position along the spiral) and fell well short on the one they could not (height) — a difference this book can put a number on that the original two papers, comparing their methods qualitatively, did not. Spectral clustering's Laplacian eigenvectors turned a clustering problem k-means could not solve at all into one it solved perfectly, exactly by the mechanism Belkin and Niyogi and von Luxburg describe: the eigenvectors are computed once, and an old, simple clustering rule is then trivially adequate on the new representation. And Isomap's one named computational limit — Floyd's algorithm's O(N³) shortest-path cost — turns out to have two separable parts once tested directly: the wall-clock cost the 2000 paper's authors actually worried about is gone, an artifact of a slower era's choice of shortest-path routine rather than of the method itself, while the quadratic memory cost of the distance matrix the eigendecomposition needs is not an artifact of anything and persists at whatever scale of hardware is brought to it. The chapter's four methods, reduced to source, share exactly one mathematical object — a quadratic form in unknown coordinates, minimized subject to a normalization constraint, solved by taking extreme eigenvectors of a matrix built from local neighborhoods — and differ only in which local relationship that matrix is built to preserve. The reruns show that difference is not a difference of degree, readable off asymptotic complexity or engineering effort, but a difference of kind, readable off which quantity a mid-sized rerun on a laptop actually recovers and which it does not.
By the late 1990s a mismatch had become impossible to ignore. The supervised paradigm that had carried machine learning through the previous two chapters of this book assumes a training set of (input, label) pairs large enough to pin down a hypothesis — but on the growing web, in medical records, in speech archives, raw instances were arriving far faster than anyone could annotate them. Blum and Mitchell put the asymmetry plainly: in tasks like sorting a university's web pages by topic, "unlabeled examples are significantly easier to come by than labeled ones," since a crawler can gather the pages while a label still requires a human to look at each one and decide. Nigam, McCallum, Thrun and Mitchell, working on newsgroup and newswire text rather than web pages, made the same point from the labeler's side of the desk. Lang (1995) had found that a person needed to read and label about a thousand Usenet articles before a classifier reached even 50% precision, and, as Nigam et al. observed, "most users of a practical system, however, would not have the patience to label a thousand articles—especially to obtain only this level of precision." Meanwhile thousands of unlabeled articles sat freely available on the same newsgroups for the taking. The question this chapter takes up is whether those free, label-free examples can be made to do useful statistical work. The answer is not obviously yes. An unlabeled point tells a learner nothing by itself about which side of a decision boundary it falls on; as Nigam et al. put it, "unlabeled data alone are generally insufficient to yield better-than-random classification because there is no information about the class label." What unlabeled data does carry is information about the shape of the input distribution — where the mass of examples sits, how densely they cluster, which points are near which. Whether that geometric information can be converted into label information depends on whether an assumed model structure — a mixture of Gaussians, a low-dimensional manifold, a graph of pairwise similarities — actually matches the data, not on how many labels are supplied. When the match is good the effect can be large: Nigam et al. report that on one newsgroup classification task, a traditional learner needed 2000 labeled examples to reach 70% accuracy, while their EM-based method reached the same accuracy from only 600 labeled examples together with 10,000 unlabeled ones — "the technique reduces the need for labeled training examples by more than a factor of three." With only 40 labeled documents, two per class, adding unlabeled data raised accuracy "from 27% to 43%." That gain, and the failure modes that appear when the structural assumption is wrong, are what the rest of this chapter works out in detail — first for the co-training and generative-model approaches sketched here, then, in the sections that follow, for the graph-based transductive methods that recast the whole problem as inference over a similarity graph joining every labeled and unlabeled point at once.
Blum and Mitchell's co-training gave the first of the two approaches this chapter follows through in detail, and it works by an assumption on the *features* rather than on the data's density. Suppose each instance can be split into two views, x = (x1, x2) — for a web page, one view is the words occurring on that page, the other the words occurring in hyperlinks that point to it. The method assumes "either view of the example would be sufficient for learning if we had enough labeled data," and additionally that the two views are not too redundant with one another — that they are compatible but, in the paper's own summary of the model, "redundant but not completely correlated." Given those two assumptions, the algorithm trains a separate classifier on each view using the small labeled set, then, per the paper's Table 1, allows each classifier to label p positive and n negative examples drawn from a pool U' of unlabeled examples, adding those self-labeled examples to the shared labeled set before retraining both classifiers and replenishing the pool: a page-based classifier and a hyperlink-based classifier are trained independently, each labels a fixed small number of new positive and negative examples from the pool, and those self-labeled examples are folded into the shared labeled set before the next round of training. Each view, in effect, manufactures labeled data for the other. Blum and Mitchell paired this algorithm with a PAC-style analysis of when it must work, then tested it on the task of classifying university web pages as course home pages or not, training naive Bayes classifiers from only 3 positive and 9 negative labeled examples plus a large unlabeled pool. Supervised training on the labeled set alone gave error rates of 12.9% for a page-based classifier, 12.4% for a hyperlink-based classifier, and 11.1% for the two combined; co-training on the same labeled set plus the unlabeled pool drove those down to 6.2%, 11.6%, and 5.0% respectively — the page-based and combined classifiers cutting their error roughly in half. The authors were candid that the two-view independence assumption is an idealization: "even for the optimal pair of functions" drawn from the two hypothesis classes, inconsistent examples on which the two views disagree are to be expected, and they left open "to what extent the consistency constraints in the model" and "the mutual independence" assumption between the views "can be relaxed and still allow provable" guarantees. The method's bite is conditional on a structural fact about the domain — that it splits into two sufficiently independent, sufficiently redundant views — not on the sheer quantity of unlabeled data assembled.
Nigam, McCallum, Thrun and Mitchell took the other route: instead of splitting the features into two views, they committed to a single generative story for the whole document and let the unlabeled data help fit it. The setting is naive Bayes recast as a mixture model: "every document is generated according to a probability distribution defined by a set of parameters," with a mixture component c_j selected according to class prior P(c_j | θ) and the component then generating the document's words according to P(d_i | c_j; θ). The move that licenses using unlabeled data is the assumption of "a one-to-one correspondence between mixture components and classes" — if that holds, a document's label is just the name of the component that generated it, and estimating the mixture's shape is a task unlabeled data can do on its own. Fitting the mixture is exactly what Expectation-Maximization was built for: "the algorithm first trains a classifier with only the available labeled documents, and uses the classifier to assign probabilistically-weighted class labels to each unlabeled document by calculating the expectation of the missing class labels. It then trains a new classifier using all the documents—both the originally labeled and the formerly unlabeled" — and the loop repeats. The theoretical warrant for expecting this to help comes from a simpler case the authors use as intuition: a mixture of two Gaussians, one per class. "It is well known that unlabeled data alone, when generated from a mixture of two Gaussians, are sufficient to recover the original mixture components," and once the component shapes are known, labeling which Gaussian is which class is "known to converge exponentially quickly in the number of labeled samples." Unlabeled data locates the clusters; a handful of labels then only need to say which cluster is which. Ported to text with the naive Bayes mixture, the empirical payoff was substantial: on 20 Newsgroups, "classification error is reduced by 30% when trained with 300 labeled and 10000 unlabeled documents," and — reprising the number the setup section already previewed — 600 labeled documents together with 10,000 unlabeled ones matched the accuracy that a labeled-only learner needed 2000 labeled examples to reach. But the authors were explicit that the whole argument is hostage to an assumption that "almost certainly is untrue in real-world domains such as text classification": that documents really are generated by one mixture component per class. When that one-to-one correspondence fails — when a "sports" class is really two sub-topics, hockey and baseball, each with its own word distribution — EM has no way to know it, and folding in unlabeled data "may actually degrade rather than improve classifier accuracy." The chapter's next task is to see how these two families — co-training's two independent views, and EM's single generative mixture — compare with a third strategy that assumes neither: one built directly on the geometry of a similarity graph.
Set side by side, co-training and generative EM look like the same wager placed on different tables. Both convert free unlabeled examples into label information only by smuggling in a structural assumption that cannot itself be checked from unlabeled data alone: co-training needs the instance to factor into two views that are individually sufficient and jointly not too correlated; EM needs the instances to have been produced by a mixture with one component per class. Neither assumption is testable by simply looking at more unlabeled points — more web pages do not tell you whether the page-text view and the hyperlink-text view are independent enough, and more newsgroup postings do not tell you whether "sports" is really one cluster or secretly two. Both papers reported large gains when the assumption approximately held (co-training roughly halving error on course-page classification; EM cutting labeling requirements by a factor of three or more on 20 Newsgroups) and both reported, or anticipated, degradation when it did not. This is the pattern the rest of the chapter will keep testing: every method for exploiting unlabeled data purchases its leverage with an assumption about how the unlabeled points relate to the labeled ones, and the interesting empirical question for any new dataset is never how much unlabeled data is available but whether that dataset's geometry happens to match the assumption on offer. The graph-based methods taken up next make this trade explicit by replacing "two views" or "one mixture component per class" with a third, more general commitment — that nearby points, measured by some similarity graph over the entire pool of labeled and unlabeled examples together, are likely to share a label — and by turning label propagation over that graph into a problem of harmonic functions solvable in closed form.
The graph-based approach folds every labeled and unlabeled example into one object: a weighted graph G on n = l + u vertices, the first l corresponding to the labeled examples x1,...,xl with labels y1,...,yl, and the remaining u to unlabeled examples. An edge weight wij, typically wij = exp(−‖xi−xj‖²/2σ²) or restricted to k nearest neighbors, encodes how strongly the graph "believes" xi and xj should share a label. Collect the weights into a matrix W ∈ Rn×n, the degrees into D := diag(W1n), and form the graph Laplacian — the combinatorial form LC := D − W, or the normalized form LN := In − D⁻¹/²WD⁻¹/²I. Both are positive semidefinite, so for any vector f ∈ Rn assigning a real value to every vertex, f⊤Lf ≥ 0, and, writing it out, f⊤LCf = ½ Σi Σj wij(f(xi) − f(xj))². The quantity is exactly a smoothness penalty: it is small precisely when f takes similar values on strongly-connected vertices, whatever f's values on weakly-connected ones. This is the graph's operationalization of the assumption the previous section left implicit — that nearby points, now measured by the similarity graph rather than by two independent views or one generative mixture, are likely to share a label. The task is then framed, following Zhu, Ghahramani and Lafferty's original 2003 formulation as it is restated in later surveys of the method built on it, as an energy minimization over a real-valued (or, for c classes, matrix-valued) function F on the vertices, constrained to agree with the known labels: writing the training sample so that the first l rows of F are pinned to the label matrix YL, the graph SSL problem is
min_{F∈R^{n×c}} tr(F⊤LF) subject to F_L = Y_L — this is Zhu, Ghahramani and Lafferty's 2003 formulation of graph-based transduction, as it is restated in the later literature this section draws on, where F ∈ R^{n×c} is a soft-label matrix over all n = l + u vertices and c classes, F_L its submatrix of rows on the l labeled vertices, and Y_L ∈ {0,1}^{l×c} the corresponding one-hot label indicator. The trace form tr(F⊤LF) is just Σj F_{·j}⊤L F_{·j}, the smoothness penalty of the previous block applied separately to each class's soft-label column. "Since L ≽0, we obtain the following closed-form solution" — the objective is convex, so partitioning L into blocks L_LL, L_LU, L_UL, L_UU conformally with F's labeled/unlabeled split, holding F_L = Y_L fixed reduces the problem to the unconstrained quadratic tr(F_U⊤L_UU F_U) + 2 tr(F_U⊤L_UL Y_L) in the free block F_U alone. Differentiating and setting the result to zero gives the stationarity condition L_UU F_U + L_UL Y_L = 0, i.e. F_U = − L_UU⁻¹ L_UL Y_L, invertible whenever every unlabeled vertex is connected by some path to a labeled one. Written out coordinate by coordinate for the combinatorial Laplacian, this closed form says exactly that at every unlabeled vertex i, F(xi) = Σ_j w_ij F(xj) / Σ_j w_ij — the value at an unlabeled point is the weighted average of its neighbors' values, whether those neighbors are themselves labeled or not. A function with that mean-value property at every free vertex, fixed only at a boundary set, is precisely a harmonic function in the discrete sense — the graph analogue of solving Laplace's equation with Dirichlet boundary data, here on the empirical graph the sample defines rather than on a continuous domain. "Due to the harmonic property," the resulting F is non-negative and, when L is the combinatorial Laplacian, each row sums to one, so that F_ij can be read as a posterior probability that vertex i carries label j — a soft classification falling directly out of solving a linear system, with no iterative training loop and no likelihood to maximize.
The closed form is exact, but on a large graph inverting L_UU directly costs O(u³), so the same solution can be reached by relaxation instead of matrix inversion. "Let P := D⁻¹W ∈ Rⁿˣⁿ be the probability transition matrix" of a random walk on the graph — the walk that, from any vertex, moves to neighbor j with probability proportional to w_ij. Partition P the same way as L, into blocks P_UU and P_UL, and starting from any initial guess F_U^(0), iterate F_U^(t+1) = P_UU F_U^(t) + P_UL Y_L: at each step every unlabeled vertex updates its soft label to the P-weighted average of its neighbors' current soft labels, with labeled neighbors contributing their fixed one-hot values. "This converges to" F_U* = (I_u − P_UU)⁻¹P_UL Y_L, which is the same closed-form solution reached by matrix inversion, since P_UU = D_UU⁻¹W_UU and a short calculation shows (I_u − P_UU)⁻¹P_UL = −L_UU⁻¹L_UL for the combinatorial Laplacian. The iterative form makes the underlying process legible: label information does not act at a distance, it diffuses outward from the labeled vertices one hop at a time, absorbing at each labeled vertex and averaging at each unlabeled one, until the whole graph settles into a fixed field of soft labels — hence "harmonic functions" and "Gaussian fields" in the method's name, and hence its natural description as a random walk with the labeled points as absorbing boundary states. The hard constraint F_L = Y_L, however, assumes the given labels are noise-free — a single mislabeled training point is propagated with full confidence to its entire neighborhood. Zhou et al.'s Local and Global Consistency algorithm relaxes exactly this, as a later paper restates it: "A widely used method for graph-based SSL is the Local and Global Consistency (LGC)... LGC can be formulated as a convex optimization problem that balances fitness on labeled examples and smoothness on the graph through a Laplacian regularizer term" — concretely, replacing the hard constraint with a quadratic penalty for disagreeing with the given labels, min_F tr(F⊤LF + (F−Y)⊤Σ(F−Y)), where Y now has zero rows for the unlabeled vertices and Σ is a diagonal matrix of per-vertex confidence weights. Differentiating and setting the gradient to zero gives F = (L + Σ)⁻¹ΣY directly, in one step, without partitioning into labeled and unlabeled blocks at all. "In the original formulation, we have Σ = µI_n where µ ∈ R*_+ is the 'global' regularization parameter" — a single scalar trading off smoothness against fit to every given label uniformly, rather than treating the labeled set as ground truth to be matched exactly. As the confidence on labeled rows is pushed to infinity and on unlabeled rows to zero, the LGC solution converges back to the hard harmonic solution of the previous block — the Gaussian-fields method is the µ → ∞ limit of the softer, more robust regularized one, not a different assumption but a different point on the same trust-in-labels dial.
Both the hard-constrained harmonic solution and its soft-penalty LGC relaxation share a further silent assumption: that the labeled seeds are a fair sample of how classes are distributed across the whole graph. When they are not — when the graph is nearly disconnected into class-pure clusters of very different size, or the labeled points happen to concentrate on one side of an otherwise balanced structure — the harmonic solution can degenerate, since nothing in tr(F⊤LF) subject to F_L = Y_L stops almost all of the propagated mass from settling on a single dominant class merely because that class's cluster is easier for a random walk to reach. Liu and Chang's Robust Multi-class Graph Transduction (RMGT) patches this by adding two further constraints on top of the GFHF objective, as later restatements of the algorithm give them: "(1) F1c = 1n; and (2) F⊤1n = nω where ω ∈Rc can be the class prior probabilities or the uniform class distribution (ω = 1c/c)." The first, F1_c = 1_n, is a row-stochastic constraint already implicit in the harmonic solution's soft-probability reading of each row of F; the second is new and does the real work. F⊤1_n sums each column of F down all n vertices, giving the total predicted mass assigned to each of the c classes across the whole graph; setting this equal to nω pins those column sums to n times a chosen prior ω — the empirical class frequencies among the labeled points, or, in the absence of better information, the uniform distribution 1_c/c. Where GFHF and LGC only constrain or penalize disagreement at the labeled vertices, RMGT additionally constrains the aggregate behavior over the unlabeled ones, ruling out solutions that are locally harmonic everywhere but globally lopsided. The closed form this produces is structurally the same object as before — an inverse of the unlabeled block of the Laplacian applied to the labeled data, plus a correction term enforcing the two normalizations — so the added robustness costs no new machinery, only two more linear constraints folded into the same quadratic program.
Every derivation so far has treated f⊤Lf as a smoothness penalty by fiat — small when f agrees on well-connected vertices, without asking what "well-connected" is a discrete stand-in for. Belkin and Niyogi's convergence analysis supplies the missing justification, and in doing so explains why the graph Laplacian is the right heuristic rather than merely a convenient one. Their starting observation is that the object every graph-based method regularizes against is not principled on its own terms: "in almost all cases, the graph itself is an empirical object, constructed as it is from sampled data. Therefore any graph-theoretic technique is justifiable only when it can be related to the underlying process generating the data." The process most of these methods implicitly assume is that the data — the vertices of the graph — are samples from some low-dimensional manifold M embedded in the ambient feature space, and that the quantity one actually wants to keep small is not the graph sum but the manifold's own Dirichlet energy ∫_M |∇f|² for a function f defined on M, the continuous analogue of the discrete f⊤Lf. Their central result closes this gap: "we show that for the uniform distribution on the manifold, the graph Laplacian converges to the Laplace–Beltrami operator as the number of points increases and the kernel bandwidth is selected appropriately" — that is, under uniform sampling of the manifold and a bandwidth t_n in the edge weights w_ij = exp(−‖x_i−x_j‖²/t_n) shrunk toward zero at a controlled rate as n grows, the empirical graph Laplacian L_n applied to a smooth function sampled at the data points converges, in the limit, to the Laplace–Beltrami operator Δ_M applied to that function on the manifold — the same operator whose associated quadratic form ∫_M f Δ_M f is the Dirichlet energy. This is what promotes f⊤Lf from an ad hoc similarity-graph heuristic to a discretization: on a genuine low-dimensional manifold, densely enough sampled, minimizing the graph penalty is minimizing an estimate of the manifold's own smoothness functional, and the harmonic solutions of Zhu, Ghahramani and Lafferty, the regularized solutions of Zhou et al., and the constrained solutions of Liu and Chang are all, to first order, computing the same continuous object by different finite-sample routes. The convergence is asymptotic and depends on assumptions — uniform density, an appropriately shrinking bandwidth, a genuine underlying manifold of the assumed dimension — that a finite, arbitrarily-distributed, possibly non-manifold dataset need not satisfy; the theorem bounds how good a heuristic the graph Laplacian is, it does not remove the heuristic. What it does remove is the sense that the whole graph-transduction apparatus of this section floats free of any object outside the sample. With the derivation complete — energy, closed form, iterative propagation, robustness constraint, and asymptotic justification all in hand — the next section puts the harmonic solution back to work on data, to see how much of this machinery survives contact with a graph built by a nearest-neighbor rule on points that only approximately live on a manifold.
The three methods derived in this chapter — co-training's two independent views, EM's single generative mixture, and the graph Laplacian's harmonic propagation — each make a falsifiable claim: given a labeled set too small to learn from alone, adding unlabeled data under the right structural assumption should close much of the gap. None of the original benchmarks survive intact in an offline corpus thirty years on: Blum and Mitchell's WebKB course-page corpus, with its page-text and hyperlink-anchor-text views, is not bundled with any standard library; Zhu, Ghahramani and Lafferty's original two-moons and USPS-derived digit benchmarks likewise are not. What follows reruns the three algorithms exactly as derived above — the co-training loop of section 1, the EM-augmented naive Bayes of section 1, and the closed-form F_U = −L_UU⁻¹L_UL Y_L of section 2 — against modern, freely available stand-ins: 20 Newsgroups postings split into a header view and a body view in place of page and hyperlink text, and sklearn's bundled digit images in place of the SSL-book digit benchmark. Every number below comes from code executed for this book, not from the original papers; where the rerun's numbers diverge from the papers', that divergence is reported rather than smoothed over.
The first rerun implements Nigam et al.'s EM-augmented naive Bayes exactly as the multinomial mixture of section 1: a class-conditional word model fit first from labeled documents alone, then refit by alternating an E-step — computing P(c|d) for every unlabeled document from the current parameters — with an M-step that re-estimates class priors and word probabilities from the labeled documents (held at their true class) together with the unlabeled documents (weighted by their current soft posterior). The rerun uses sklearn's bundled 20 Newsgroups corpus, all twenty classes, vectorized to a 3,000-word vocabulary with a Laplace-smoothed multinomial likelihood, exactly the model in the closed-form update the paper derives. One deliberate divergence from the original: headers, footers and quoted reply text are stripped before vectorizing, because the raw 20 Newsgroups headers (an explicit "Newsgroups:" line naming the source group) leak the label directly — a shortcut the 1999 paper's raw-text pipeline may not have guarded against as carefully as modern practice does, meaning this rerun's supervised baseline is handicapped relative to the original's, not helped. Three labeled-set sizes were tried, each paired with a capped 4,000-document unlabeled pool drawn from the same training split: with one labeled example per class (20 documents total), pure supervised naive Bayes reached 13.4% test accuracy, and the same 20 labeled documents plus EM over the unlabeled pool reached 23.7% — a ten-point absolute gain. With ten labeled examples per class (200 documents), supervised accuracy was 30.6% and EM-augmented accuracy 36.4%, a gain of 5.8 points. With thirty labeled examples per class (600 documents), the two nearly converge: 41.0% supervised versus 42.6% with EM, a 1.6-point gain. The pattern reproduces the shape of Nigam et al.'s own result even though the absolute numbers differ (their reported baseline was a two-class course-page task reaching 70% accuracy from 2,000 labeled examples, this rerun a twenty-class, twenty-category task with a stripped, harder feature set): the benefit of unlabeled data is largest exactly where labeled data is scarcest, and shrinks toward zero as the labeled set grows large enough to pin down the mixture on its own — because at that point EM's structural assumption is doing no work that the labels were not already doing.
The second rerun implements Blum and Mitchell's co-training loop verbatim from its Table 1 description in section 1: two naive Bayes classifiers trained on separate views of the same tiny labeled seed, each allowed to add its p most-confident positive and n most-confident negative predictions from a refreshed pool of unlabeled examples to the shared labeled set every round. Lacking the WebKB course-page corpus, the rerun substitutes 20 Newsgroups postings with headers retained, split at the first blank line into a header view (From/Subject/Organization/Newsgroups lines — playing the role the original's hyperlink-anchor-text view played, a short, class-correlated fragment attached to the document rather than drawn from its main content) and a body view (the message proper, playing the role of the original's page-text view). The classification task was built to match the original's skew: "soc.religion.christian" as the rare positive class (25.3% base rate, close to the original course-page rate) against three other newsgroups pooled as negative, with a seed of 3 labeled positive and 9 labeled negative documents — the exact numbers Blum and Mitchell used — against a 2,000-document unlabeled pool, run for thirty co-training iterations with p=1, n=3 self-labeled examples added per view per round, matching the original's parameters. Trained on the 12-document seed alone, the header-view classifier reached 23.9% test error, the body-view classifier 9.2%, and their averaged-probability combination 15.8%. After thirty rounds of co-training, with the labeled set grown by self-labeling to 238 documents, the same three numbers fell to 17.1%, 3.1%, and 2.5% respectively. The direction and rough proportion of the effect match Blum and Mitchell's own result — their page-based classifier fell from 12.9% to 6.2% error and their combined classifier from 11.1% to 5.0%, both roughly halving — even though the absolute error rates differ because the task, the two views, and the base classifier's feature set are all different from the original's. What is reproduced is the qualitative claim the paper is actually making: that two independently-trained views, each bootstrapping labels for the other from a shared unlabeled pool, can drive a combined classifier's error down by an order of magnitude from a labeled seed too small for either view to learn from on its own — provided, as section 1 stressed, the two views really are each informative and not simply redundant restatements of the same signal.
The third rerun implements the closed form derived in section 2 directly: build a k-nearest-neighbor similarity graph with Gaussian edge weights, form the combinatorial Laplacian L = D − W, partition it into labeled and unlabeled blocks, and solve F_U = −L_UU⁻¹L_UL Y_L exactly as written there — no iteration, no likelihood, a single linear solve. In place of the original digit benchmark from the SSL-book suite (referenced in section 2 as "DIGIT-1," a 1,500-point, two-class subset used in the later restatement this chapter draws on), the rerun uses sklearn's bundled digits data set — 1,797 8×8 handwritten-digit images across all ten classes, a harder multi-class version of the same task, built here from a k=10 nearest-neighbor graph with the kernel bandwidth set by the median pairwise-distance heuristic. At one labeled example per class (10 labeled points total, out of 1,797), the harmonic solution classified the remaining 1,787 points at 92.2% transductive accuracy. A supervised baseline given the identical ten labeled points — a 1-nearest-neighbor classifier trained on nothing else — reached 71.3%: a twenty-point gap attributable entirely to the unlabeled points' geometry. At three, five, and ten labeled examples per class the harmonic solution reached 90.3%, 92.2%, and 96.1%, against the 1-NN baseline's 80.8%, 83.4%, and 89.7% — the gap narrows as labels accumulate but does not close by ten per class. As a sanity check, the same splits were also run through scikit-learn's own LabelSpreading implementation (RBF-weighted graph, soft label-clamping rather than the hard Dirichlet boundary of the exact harmonic solution): it scored 83.7%, 87.1%, 89.8%, and 92.9% at the same four label budgets — better than the naive 1-NN baseline, as expected from the same family of method, but consistently below the closed-form solution implemented directly from section 2's derivation. That gap is itself informative: it is not evidence against label propagation but a reminder that "the graph-based method" is a family with a free parameter (here, LabelSpreading's clamping strength) that the hard-constrained harmonic solution sets to its limit, and that limit was the better choice on this data set. Across every label budget tested, the qualitative claim behind Zhu, Ghahramani and Lafferty's derivation held without exception: a supervised classifier restricted to a handful of labeled points is reliably beaten by the same points used only to pin boundary values on a harmonic function computed over the full graph of labeled and unlabeled data together.
None of these three reruns needed anything a 1999 or 2003 workstation would have found remotely taxing — the harmonic solve is a single dense linear system over at most a few thousand unlabeled points, the EM loop a few dozen passes over sparse word counts, the co-training loop a few dozen retrainings of naive Bayes on a labeled set that never exceeds a few hundred documents. This chapter's limitation was never computational; it is the one the central question of this book asks after: would the bound still hold with a 2026 datacenter behind it? Here the answer is unambiguously yes, because the bound was never a hardware ceiling to begin with. Every gain measured above was purchased by an assumption about how the unlabeled data relates to the labeled data — two sufficiently independent views, one generative mixture component per class, a similarity graph on which nearby points really do share labels — and every degradation the original papers anticipated when that assumption failed is a mathematical fact about mismatched structure, not a fact about clock speed or memory. Running the exact same three algorithms today, on a laptop, against harder and larger substitute benchmarks than the originals used, reproduces both halves of that finding: substantial, sometimes order-of-magnitude gains when the assumption approximately holds (co-training's error roughly halving, the harmonic solution beating a supervised baseline by twenty points at the smallest label budget), and a gain that shrinks toward nothing as the labeled set grows large enough that the unlabeled data's structural assumption stops doing any work the labels were not already doing. A faster machine changes how large a graph or a vocabulary can be handled; it does not change whether the graph's geometry matches the labels, or whether two views are really independent, or whether a class is really one mixture component. That is this chapter's version of the book's recurring separation between a limitation of the era and a limitation of the mathematics — and on the evidence rerun here, semi-supervised learning's central claims sit entirely on the mathematics side of that line.
This book was drafted, revised, and fact-checked by an event-sourced multi-agent system — an author, an adversarial critic, a planner, and an arbiter — working over a corpus of ~5,000 primary papers, directed and assembled by its human author. Nothing here rests on trust in the process: the process is published. Every sentence traces to logged reads of the primary sources; every quotation was machine-verified verbatim against the paper it cites; every number in an experimental rerun comes from an executed, independently reproduced computation; and the full construction record — the tape — replays to exactly this text.
| Words | 176,659 |
| Sections | 117 |
| Construction events on the tape | 13,539 |
| Executed experiments | 398 |
| Critiques filed and resolved | 175 |
Preprint, text only — figures are forthcoming. Source, tape, lab, and harness: github.com/doInfinitely/long-detour.
Even the jacket keeps receipts: the cover concepts and the design transcript are in the repository, including the quote check that caught a mocked-up cover inventing archival telemetry. Jacket copy passes the same quote check as the prose.