THE LEARNING PATH · Intermediate
Mechanistic interpretability of LLMs: a practical guide
Learn how features, circuits and causal interventions help investigate an LLM—and what these methods still cannot prove.
What mechanistic interpretability tries to explain
A language model maps a sequence of tokens to scores for the next token. Mechanistic interpretability asks a more demanding question than “which input mattered?”: which internal computations produced a behavior, and how can we test that account? The objective is not to turn every vector into a sentence. It is to build explanations at the level of representations, components and interactions, then challenge them with controlled experiments.
This distinguishes the field from a generic explanation interface. An attention map, a probe or a convincing verbal story can reveal a pattern, but a pattern alone does not establish that the model used it. A useful investigation moves from observation to a hypothesis, then to an intervention and an independently chosen measurement.
From representations to features and circuits
An activation is a value computed for one input at one location in the network. A feature is a direction or pattern that tracks something useful across examples. It need not correspond to one neuron: models can distribute a feature across many coordinates, and several features can share the same representational capacity. This phenomenon is often discussed as superposition.
A circuit is a proposed collection of components and connections that performs part of a computation. Attention heads, multilayer perceptrons and residual-stream directions may all participate. Naming the pieces is not enough. A circuit claim should specify the task, the sites, the information transmitted, the expected effect of an intervention and the examples on which it was validated.
The main families of methods
- Probes test whether information can be decoded from an activation. They establish availability, not necessarily use.
- Logit lens and attribution methods connect internal states to output scores. They help localize candidate contributions but depend on the decomposition and measurement chosen.
- Activation patching transfers a state from one run to another and measures whether a defined behavior is restored. It provides causal evidence for the intervention under the tested conditions.
- Ablation removes or replaces a component. A change can show necessity in a particular setting, while redundancy can hide a real contribution.
- Sparse autoencoders learn a larger set of sparse features from activations. They can make recurring patterns easier to inspect, but feature labels and causal relevance still require evaluation.
- Circuit discovery searches for a smaller computational subgraph. Automated procedures make the search scalable, not automatically faithful.
A controlled activation-patching example
Start with two prompts that differ in one behaviorally relevant way. Run both, choose a token position and layer, and save the activation from the clean condition. During the corrupted run, replace only that activation and continue the forward pass. Measure a contrast fixed in advance, such as the logit difference between two candidate tokens.
If the intervention restores the contrast, the patched state played a causal role in this protocol. It does not follow that the vector contains one human-readable concept or that the full circuit has been found. Self-patching, reverse patching, matched perturbations and held-out examples help expose implementation errors and selection effects. The site’s guided activation-patching lesson develops this protocol step by step.
What a result proves—and what it does not
Mechanistic evidence is conditional. It depends on the model, prompts, tokenization, sites, replacement rule and metric. A positive result supports a causal statement about that intervention. A null result may indicate irrelevance, a poorly chosen site, a weak metric or a redundant pathway. A large effect does not by itself make the explanation complete.
Strong work therefore reports controls, negative cases, the search procedure and validation on examples not used to select the mechanism. It also distinguishes a teaching simplification from an experiment on an actual model.
A reproducible workflow
- Define the behavior and metric before inspecting internal states.
- Construct comparable clean and corrupted inputs.
- Observe candidate representations without calling them causes.
- Intervene at explicit sites with documented code and model versions.
- Compare against controls and alternative hypotheses.
- Validate on held-out prompts and report failures.
- State the narrowest conclusion supported by the evidence.
Useful primary resources include TransformerLens patching utilities, Toy Models of Superposition and the ACDC circuit-discovery paper.
Where to go next
Use the Lab to practice separating a result from its interpretation. Volume II, The Internal Mechanics of LLMs, develops the broader experimental approach and its limits.