Chapter 20 Deep learning and neural scaling
10 minute read
20.1Universal function approximation and backpropagation
For the first four decades of computer science, pioneers of artificial intelligence tried to build thinking machines from the top down. Researchers assumed that writing down explicit logical rules of human knowledge would suffice to create general intelligence. That symbolic programme triumphed in tidy, rule-bound worlds such as checkers, algebraic integration, and grandmaster chess. Yet it shattered when a machine tried to recognize a face in dim light, walk across a room, or grasp common sense.
In the 1980s, the roboticist Hans Moravec diagnosed why this failure occurred. High-level abstract reasoning, such as formal logic or calculus, is an evolutionary newcomer barely a few thousand years old. It requires very little raw computation. By contrast, effortless biological perception and motor control rest on hundreds of millions of years of natural selection. Those capacities run on vast parallel neural circuits shaped by survival. What feels easy to a human toddler is computationally immense. What feels hard to a mathematician is a thin symbolic veneer atop an ancient sensorimotor ocean.
To cross that ocean, a rival scientific tradition took direct inspiration from the biological synaptic networks of Book V. Warren McCulloch and Walter Pitts proposed the first mathematical model of a neuron in 1943. Fifteen years later, Frank Rosenblatt built the perceptron to learn weights from examples. Rather than following hand-written rules, an artificial neural network adjusts its internal connection strengths until its outputs match the world.
Mathematically, a deep neural network is a parameterized function controlled by a vast vector of adjustable connection weights. It transforms an input vector of numbers by passing it through a sequence of layers. Each layer multiplies the incoming vector by a matrix of weights, adds a vector of bias offsets, and applies an element-wise nonlinear activation function to every coordinate.
A standard choice for that activation is the rectified linear unit, defined by . This simple threshold passes positive numbers unchanged and resets negative numbers to zero. Why is that nonlinear bend necessary at every layer? Without it, multiplying any number of linear matrices in sequence collapses into a single flat matrix. A purely linear network of a thousand layers can compute nothing more than a single layer.
Once a non-polynomial activation sits between the matrices, the expressive power of the system changes completely. Between 1989 and 1993, George Cybenko, Kurt Hornik, and Moshe Leshno proved the universal approximation theorem. They showed that a neural network with a non-polynomial activation can approximate any continuous function on a closed and bounded domain to arbitrary accuracy. Stacking many layers into a deep hierarchy lets each layer compose features from the layer below it. Early layers detect simple edges or sound frequencies, while middle layers assemble parts and motifs. Deep layers then represent whole objects and abstract concepts using exponentially fewer parameters than a shallow network.
Yet universal expressive power means nothing if we cannot find the right weights among a trillion possibilities. Suppose we present the network with training examples and measure its prediction error with a differentiable loss function . For probabilistic predictions, engineers minimize the cross-entropy loss, the negative log-probability assigned to the true outcome. As we saw in Chapters 14 and 16, minimizing cross-entropy is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution and the model.
How can an algorithm adjust one trillion weights at once to walk downhill on this high-dimensional loss surface? Imagine trying to estimate the slope of the loss surface by naive forward perturbation. Suppose we nudged each weight one by one to see whether the error rose or fell. Measuring a single gradient step for one trillion weights would require one trillion separate forward runs of the network. Training a modern foundation model would take longer than the age of the universe.
The mathematical breakthrough that made large-scale learning possible is reverse-mode automatic differentiation, known in neural networks as backpropagation. Seppo Linnainmaa formulated the general reverse-mode algorithm in 1970, and Paul Werbos applied it to neural networks in 1974. David Rumelhart, Geoffrey Hinton, and Ronald Williams then demonstrated its power for learning internal representations in 1986. Backpropagation applies the multivariable chain rule in reverse from the scalar output loss. Defining the error sensitivity vector at each layer as the derivative of the loss with respect to that layer's pre-activation inputs, the algorithm propagates blame backward down the network:
By reusing the intermediate activations saved during the forward pass, backpropagation finds the exact analytical gradient for all one trillion weights at once. A calculation that would take a trillion forward runs by perturbation finishes in a single forward and backward sweep.
Even with backpropagation in hand, training networks deeper than 20 layers remained notoriously unstable for decades. Multiplying dozens of transposed weight matrices in sequence causes the backward error signal either to vanish exponentially toward zero or to explode. In 2015, Kaiming He and colleagues removed this barrier by introducing residual connections. Instead of building a new vector from scratch, each block adds a learned update onto the input: .
When we differentiate this sum with respect to an earlier layer, the Jacobian matrix gains an explicit identity matrix. Because of that identity term, gradients flow backward without attenuation along a shared communication highway called the residual stream. Each layer reads from the residual stream, computes a refinement, and writes its update back into the stream. Networks of hundreds of layers can then train smoothly from end to end.
20.2Attention and the Transformer architecture
Before 2017, neural networks processed sequential streams such as human language, computer code, and genomic sequences using recurrent loops. A recurrent network read one word at a time and updated a single hidden memory vector from step to step. That sequential design suffered from two severe flaws. Because each step had to wait for the previous step to finish, training could not be parallelized across processors. Moreover, information from early paragraphs faded away as it was squeezed through a single fixed-size vector bottleneck.
In 2017, Ashish Vaswani and colleagues published 'Attention Is All You Need' and discarded recurrence altogether. Their replacement, the Transformer architecture, allowed every position in a sequence to communicate directly with every earlier position in parallel. To see how an autoregressive Transformer turns raw text into structured thought, we can follow a sequence through its five stages.
The journey begins with tokenization and embedding. An incoming stream of text is segmented into discrete tokens drawn from a fixed vocabulary. That vocabulary typically contains between 32,000 and 200,000 sub-word pieces. A learned embedding lookup table converts each discrete token into a continuous vector in the initial residual stream. Because a parallel network has no built-in sense of word order, the model encodes position geometrically. Rotary position embeddings rotate pairs of vector coordinates by angles proportional to the token's position in the sequence. As a result, inner products between tokens depend naturally on their relative distance along the page.
Second, before entering any processing block, each token vector passes through root-mean-square layer normalization. This step divides the vector by the root-mean-square magnitude of its coordinates and scales the result by a learned gain vector. Normalization keeps activation magnitudes steady across deep stacks so that neither forward signals nor backward gradients drift out of scale.
Third comes the core routing engine of the architecture, causal multi-head self-attention. How does an isolated word vector for a pronoun like 'it' discover which noun three sentences earlier it refers to? Inside each layer, multiple parallel attention heads operate simultaneously on the normalized residual stream. Each head uses three learned matrices to project every token vector into three specialized vectors of lower dimension . A query vector represents what information a token seeks. A key vector advertises what content an earlier token holds. Finally, a value vector carries the payload that an earlier token will transmit if selected.
To decide where to look, each token takes the dot product between its query vector and the key vector of every preceding token. Those compatibility scores are divided by the square root of the head dimension so that their variance stays near 1 as dimension grows. Without that scaling factor, large dot products would push the exponential normalization into flat regions with vanishing gradients. Adding a causal mask matrix, which sets future entries to negative infinity so a token cannot peek ahead in time, gives the attention equation:
Here the row-wise softmax function exponentiates the scaled scores and divides by their sum, turning raw compatibility scores into positive attention weights that add to 1. Physically, self-attention is a differentiable, content-addressable associative memory, mathematically dual to a continuous modern Hopfield network. In one parallel step, every token queries the preceding context and retrieves relevant syntactic and semantic values. An output matrix then projects the combined findings of all heads back into the residual stream.
Fourth, while self-attention routes context horizontally across different token positions, the subsequent feed-forward network processes features vertically within each individual token vector. Each token passes through a gated multi-layer perceptron that expands its vector into a wider intermediate space. After applying an element-wise gating nonlinearity, the layer projects the result back into the residual stream. Mechanistic interpretability studies show that these feed-forward matrices act as vast key-value memory banks. While attention heads move context between words, the feed-forward weights store learned factual associations about the world.
Fifth, after passing through all stacked blocks of attention and feed-forward refinement, the final vector at each position is normalized and multiplied by an unembedding matrix. Scaled by a sampling temperature, the softmax function converts the resulting scores into a probability distribution over the very next token. By the chain rule of probability, the joint probability of any text sequence factors into the product of these conditional next-token probabilities.
The network learns by minimizing the average negative log-probability assigned to the true next token across trillions of words of text. This objective reveals why next-token prediction is far more than surface statistical mimicry. By Claude Shannon's source coding theorem and Ray Solomonoff's theory of algorithmic complexity from Chapters 3 and 16, minimizing negative log-probability is identical to optimal data compression. To compress a physics textbook, a software repository, or a mystery novel down to its minimum description length, memorizing local word pairs is never enough. Gradient descent forces the weights to construct internal circuits, syntax trees, and causal world models of the reality that generated the text.
20.3Neural scaling laws and emergent capabilities
For decades, machine learning seemed like an unpredictable craft where no theory could foretell how a larger network would behave. Between 2020 and 2022, Jared Kaplan and colleagues and Jordan Hoffmann and colleagues uncovered a striking empirical regularity. Across more than seven orders of magnitude of training compute , the test cross-entropy loss of a Transformer follows a smooth power law. It depends predictably on both the number of non-embedding parameters and the number of training tokens:
In this empirical law, the first term represents the irreducible Shannon entropy of natural text that no predictor can eliminate. The measured scaling exponents sit close to one another near 0.34. Minimizing this power-law loss under a fixed compute budget via Lagrange multipliers defines the compute-optimal scaling frontier. Because the two scaling exponents are nearly equal, both optimal model size and optimal dataset size grow in proportion to the square root of the compute budget. Specifically, an optimal training run scales parameters and data in lockstep, feeding the network roughly 20 tokens of text for every parameter.
Even more striking is what happens inside the network as this smooth power law descends. Aggregate cross-entropy loss decreases gradually with scale. Yet accuracy on multi-step tasks—such as arithmetic, code synthesis, and formal theorem proving—stays near zero for small models and then jumps sharply at critical scale thresholds. Cooling water undergoes a sudden phase transition into crystalline ice once thermal agitation drops below a critical point. In much the same way, reducing per-step prediction error below critical thresholds lets multi-step reasoning circuits lock together across layers without compounding noise.
Chief among these emergent capabilities is in-context learning. Without changing a single weight, a scaled Transformer can learn a brand-new task on the fly from a few examples in its prompt. Mechanistic interpretability traced this skill to specialized attention circuits called induction heads. These circuits crystallize during an abrupt phase transition in training, implementing pattern completion and adaptation directly inside the forward activations of the residual stream.
Coupling pre-trained foundation models with reinforcement learning from verifiable rewards and test-time search creates a further shift in capability. A base language model emits tokens in a single reflexive stream, much like the fast, intuitive System 1 thinking described by Daniel Kahneman. By contrast, a reasoning model spends compute at inference time to generate, verify, and backtrack over extended chains of thought before answering. By searching the space of formal proofs and executable programs, silicon architectures have begun to drive the epistemic engine of Chapter 18 autonomously.