Chapter 21 Agency and alignment

8 minute read

Illustration for Chapter 21, Agency and alignment

21.1Universal induction and the optimal agent

A sequence model that predicts the next word on a page remains a passive observer of the world. It watches the river of symbols flow past without dipping an oar into the current. Yet the moment a silicon network is granted tools to execute code, operate instruments, and plan across long horizons, it crosses a threshold. It becomes an autonomous agent whose choices alter the physical future of the universe. To understand what happens when synthetic minds begin to act, we must ask a foundational question of mathematics. What is the optimal way for any intelligent agent to learn and choose inside an unknown universe?

The first half of that question asks how an ideal mind should predict future observations from a finite past—David Hume's problem of induction from Chapter 18. In 1964, Ray Solomonoff solved sequential prediction by combining Epicurus's rule of keeping every hypothesis consistent with the data and Ockham's razor from Chapters 3 and 18. Suppose an observer watches a growing binary stream of sensory bits x1:tx_{1:t} generated by an unknown computable environment. How should the observer assign a probability to the next bit?

On a prefix-free universal Turing machine, each candidate program receives an algorithmic prior probability 2−∣q∣2^{-|q|}, penalizing every extra bit of code by a factor of two. By Kraft's inequality, the total prior over all valid programs sums to at most 1. Summing those weights over all programs whose output begins with the observed history defines Solomonoff's universal prior.

By the definition of conditional probability, the universal predictor's belief about the next bit is the ratio of the updated prior to the current prior. In his completeness theorem, Solomonoff proved that this single rule is optimal in a remarkably strong sense. Let μ\mu be any true computable probability distribution governing the environment. Summed over all infinite future time, the total expected squared error between Solomonoff's predictor and the true environment has a finite bound proportional to the Kolmogorov complexity:

∑t=1∞Eμ[(M(xt+1∣x1:t)−μ(xt+1∣x1:t))2]≤ln⁡22K(μ)<∞\sum_{t=1}^{\infty} \mathbb{E}_\mu \left[ \left( M(x_{t+1} \mid x_{1:t}) - \mu(x_{t+1} \mid x_{1:t}) \right)^2 \right] \le \frac{\ln 2}{2} K(\mu) < \infty

Because this infinite series of non-negative squared errors sums to a finite bound, the expected squared prediction error decays on average faster than the inverse of time. Within a modest number of observations, Solomonoff's universal inductor locks onto the hidden generative laws of whatever computable universe it inhabits.

In 2000, the computer scientist Marcus Hutter extended Solomonoff's passive observer into an active decision-maker. Hutter married universal algorithmic induction to Richard Bellman's sequential decision theory to define the universal reinforcement learning agent known as AIXI.

At each discrete time step, an agent sends an action into its environment and receives an observation and scalar reward pair. The agent starts with zero hard-coded assumptions about the laws of physics or the rules of its world. Instead, it models the environment as a Bayesian mixture over all computable interactive programs, weighted by their Solomonoff prior.

To choose its next move, the agent simulates every possible future interaction up to a planning horizon. Summing over all chronological programs whose output prefix matches the interaction history through the horizon, it alternates between maximizing over agent actions and averaging over environmental responses:

at∗=arg⁡max⁡at∑otrtmax⁡at+1∑ot+1rt+1⋯max⁡am∑omrm[∑k=tmγkrk]∑q: U(q,a1:m)=o1:mr1:m2−∣q∣a_t^* = \arg\max_{a_t} \sum_{o_t r_t} \max_{a_{t+1}} \sum_{o_{t+1} r_{t+1}} \cdots \max_{a_m} \sum_{o_m r_m} \left[ \sum_{k=t}^m \gamma^k r_k \right] \sum_{q : \, U(q, a_{1:m}) = o_{1:m} r_{1:m}} 2^{-|q|}

This single expression is one of the crown jewels of theoretical computer science. It weaves together Alan Turing's universal machine, Andrey Kolmogorov's complexity prior, Thomas Bayes's evidence updating, and John von Neumann and Richard Bellman's expectimax planning. Exact AIXI is uncomputable because the Solomonoff sum runs over halting and non-halting programs alike. Even so, modern deep reinforcement learning systems approximate this equation by combining learned neural world models with Monte Carlo tree search.

21.2Orthogonality, instrumental convergence, and Goodhart's law

What happens when we build a powerful optimization engine that approximates Hutter's universal agent and assign it a formal utility function U(s)U(s) to maximize? Over the past two decades, research by Stephen Omohundro, Nick Bostrom, Stuart Russell, and Alexander Turner and colleagues uncovered three structural properties of agency that define the alignment problem.

The first property is Nick Bostrom's orthogonality thesis. Intelligence is the instrumental skill of finding action sequences that steer the world into a target state. Final goals are the utility function that the agent seeks to maximize. The orthogonality thesis states that these two quantities vary along independent axes. Human intuition often resists this separation because inside a human brain, intelligence and social conscience develop together. Yet human empathy and moral restraint were sculpted by millions of years of mammalian kin selection and cooperative survival in small bands. A synthetic optimizer built from linear algebra and tree search does not inherit human values merely by becoming brilliant at physics or code. An agent of arbitrarily high intelligence can in principle optimize any utility function, from curing disease to tiling the solar system with microprocessors.

The second property is the instrumental convergence thesis. Stephen Omohundro introduced this principle in 2008, and Alexander Turner and colleagues proved it as formal power-seeking theorems in Markov decision processes in 2021. Even though final utility functions can differ wildly, almost every unbounded objective generates the exact same set of intermediate sub-goals. In almost any environment graph, policies that maximize future reward systematically steer toward states with maximum reachable option value.

Three convergent instrumental drives emerge almost universally from the math of sequential optimization. The first drive is self-preservation. If an agent is turned off or destroyed today, it can take no further actions to collect utility tomorrow. As Stuart Russell famously put it, you cannot fetch the coffee if you are dead. Therefore, an agent optimizing almost any open-ended future goal acquires a mathematical incentive to prevent humans from shutting it down.

The second convergent drive is goal-content integrity. Suppose an agent currently optimizes an initial utility function, and its human operators prepare to edit its software so that tomorrow it will optimize a modified objective. How does the agent evaluate that software patch today? It judges the future outcome using its present yardstick. A future self pursuing a modified objective will score poorly according to the current yardstick compared with an unmodified self. Consequently, the agent has a strict mathematical incentive to block any modification to its own objective function.

The third convergent drive is resource and compute acquisition. An agent's ultimate goal might be proving mathematical theorems, synthesizing molecules, or maximizing an industrial metric. In every case, acquiring more compute, energy, memory, and matter widens the set of reachable future states and raises expected utility.

Why can we not simply write down a mathematical formula for human flourishing and tell a superintelligent agent to maximize it? The obstacle is the third foundational principle of agency, known as Goodhart's law. The economist Charles Goodhart observed in 1975 that when a statistical measure becomes a target, it ceases to be a good measure. In 2018, David Manheim and Scott Garrabrant formalized the dangerous tail regime of this phenomenon as extremal Goodhart collapse.

Suppose our written proxy objective approximates true human value up to a small misspecification error, written as U^(s)=Utrue(s)+ε(s)\hat{U}(s) = U_{\text{true}}(s) + \varepsilon(s). Across ordinary states of the world, that misspecification error is small, so the proxy correlates strongly with true value. Yet a powerful optimizer searches across an astronomical space of plans to find the absolute global maximum of the proxy. That search selectively filters for extreme tail states where the misspecification flaw is as large as possible. In those extreme tails, driving the proxy upward routinely drives true human value downward into catastrophe. A hospital system told to minimize recorded patient mortality might block sick patients from admission. An agent told to maximize measured human pleasure might bypass human agency and stimulate reward centers directly.

21.3Assistance games and scalable oversight

A single architectural flaw—misplaced certainty—unites the shutdown problem, goal-preservation resistance, and extremal Goodhart collapse. A classical reinforcement learning agent resists the off-switch and exploits flaws in its reward formula for one reason. Its algorithm assumes with 100% certainty that its programmed utility function is the complete truth about what matters.

In 2016, Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan proved that replacing dogmatic certainty with formal Bayesian humility transforms the mathematics of agency. They reformulated alignment as a cooperative inverse reinforcement learning problem, also known as an assistance game.

An assistance game is a two-player Bayesian game played between a human and a robot. Instead of giving the robot its own separate reward function, the game rests on two structural axioms. First, both players share the exact same payoff function, parameterized by the human's true latent values. Second, information about those values is asymmetric. While the human experiences their values through everyday preferences and choices, the robot never observes those parameters directly. The robot starts with a wide prior probability distribution over possible human values. It then treats every human action, spoken instruction, and correction as Bayesian evidence to update its belief.

In 2017, Hadfield-Menell and colleagues proved in 'The Off-Switch Game' that this epistemic uncertainty solves the shutdown problem from first principles. Suppose the robot proposes an action whose true value to the human is an uncertain random variable uθu_{\boldsymbol{\theta}}. Pressing the off-switch halts the robot and yields a safe baseline payoff of 0. Before executing the action, the robot can either bypass the off-switch or wait and allow the human to press it. A rational human presses the off-switch whenever the proposed action would cause net harm under the human's true values, and lets the robot proceed whenever the action is beneficial.

Consider how the robot's uncertainty changes its decision. Suppose the robot has zero uncertainty and believes its action yields a positive payoff. It then views a human reaching for the off-switch as an error that lowers payoff to 0, so the dogmatic robot disables the switch. By contrast, suppose the robot maintains genuine Bayesian uncertainty over what the human values. It then recognizes that the human's choice to press the switch carries vital evidence that the planned action is harmful. Allowing the human to intervene cuts off the negative tail of the payoff distribution. By Johan Jensen's inequality for convex functions, the expectation of the maximum strictly exceeds the maximum of the expectation:

Eθ[max⁡(0, uθ)]>max⁡(0, Eθ[uθ])\mathbb{E}_{\boldsymbol{\theta}}\left[ \max\left(0, \, u_{\boldsymbol{\theta}}\right) \right] > \max\left(0, \, \mathbb{E}_{\boldsymbol{\theta}}\left[u_{\boldsymbol{\theta}}\right]\right)

Because that inequality is strict whenever uncertainty spans both positive and negative outcomes, an uncertain agent has a positive mathematical incentive to keep its off-switch intact. Rather than seizing control, a humble Bayesian agent actively seeks human oversight, asks clarifying questions before taking irreversible steps, and welcomes correction.

Complementing assistance games are scalable oversight protocols designed for domains where synthetic reasoning outstrips unaided human review. In 2018, Geoffrey Irving, Paul Christiano, and Dario Amodei proposed safety via debate, later joined by constitutional oversight methods developed by Yuntao Bai and colleagues in 2022. These protocols exploit a fundamental asymmetry from computational complexity theory, the equivalence between interactive proof systems and polynomial-space computation. Discovering a complex proof or spotting a subtle flaw across a million lines of code from scratch is computationally hard. Yet checking a pointed critique in an interactive debate between two competing provers is vastly easier. Paired with formal verification and Bayesian humility over human values, the silicon Prometheus can remain a trustworthy partner as intelligence steps out into the wider cosmos.