A transformer can produce two different outputs from prompts that differ in one relevant detail, yet inspection of attention weights or hidden-state similarity does not establish which internal states actually matter for that difference. Activation patching addresses a narrower question by intervening on a forward pass: replace a selected activation with the corresponding activation from another run, then measure how the output changes.
The result is causal with respect to that intervention. It does not automatically identify a complete circuit, a unique mechanism, or a human-readable feature. That boundary is central to using patching results correctly.
A patch changes the computation, not just the observation
Consider two inputs with matched structure. A reference input produces the behavior of interest, while a perturbed input reduces or changes that behavior. Run both inputs and cache internal activations. During a new forward pass on the perturbed input, substitute one cached activation from the reference run at a chosen site.
A site can be a residual-stream vector, an attention output, an MLP output, or a more granular component, depending on the model and instrumentation. If replacing that state restores part of the reference behavior, the intervened state carries information that downstream computation can use for the measured output.
A simple score can express the effect:
patch_effect = score(patched_run) - score(perturbed_run)For a language model, score might be a target-token logit, a logit difference between two candidate tokens, or another output statistic tied to the behavior under study. The score definition matters. A patch that raises one token’s logit can still leave its rank unchanged if competing logits rise as well.
Patching is therefore different from reading an activation and assigning meaning to it. The method asks what happens when a particular state is transplanted into another computation.
The reference and perturbed inputs define the intervention
Activation patching depends heavily on the pair of inputs. If they differ in several semantic or syntactic properties, a large patch effect cannot be assigned cleanly to one of those differences.
Matched inputs reduce that ambiguity. Suppose the target behavior is retrieval of a city associated with a country. A reference and perturbed prompt can keep sentence structure fixed while changing the country token. The resulting activation difference is still not guaranteed to represent only one concept, but fewer input changes make the intervention easier to interpret.
The perturbed run also defines the baseline from which recovery is measured. Replacing an activation from a reference run into an unrelated prompt can create an off-distribution internal state. A changed output then shows sensitivity to the intervention, but its connection to the original mechanism is weaker.
This is one reason patching results should be stated in terms of the tested input pair and metric rather than as universal properties of a model component.
Patch location changes the claim
A transformer repeatedly mixes information across token positions and layers. Patching the residual stream at one token and layer replaces the combined state available at that point. Patching only an attention or MLP contribution targets a narrower part of the computation.
These interventions answer different questions.
A residual-stream patch can show that the state at a location is sufficient to transfer some behavior under the chosen conditions. It does not isolate which upstream component created the relevant part of that state. A component-output patch narrows the source, but downstream interactions can still amplify, suppress, or redirect the inserted signal.
Token position matters as well. Information associated with one input token can move into other positions through attention. A weak effect at the original token in later layers does not establish that the information disappeared; it may have been incorporated elsewhere in the sequence.
Granularity should match the claim. Broad patches are useful for locating regions of a computation. Finer patches can test candidate components after a region has been identified.
Recovery scores need a stable denominator
Some experiments report a normalized recovery score rather than a raw logit change. A common form compares the patched output with the gap between reference and perturbed outputs:
recovery =
(score(patched) - score(perturbed))
/ (score(reference) - score(perturbed))This makes 0 correspond to the perturbed score and 1 correspond to the reference score. It also introduces a failure mode: when the reference and perturbed scores are close, the denominator is small and the normalized value can become unstable.
Normalized recovery can also exceed the interval from 0 to 1. A patch may push the score beyond the reference value or in the opposite direction. Clipping such values hides information about the intervention unless clipping is part of a separately justified reporting convention.
Raw scores, the reference-to-perturbed gap, and the normalized effect together make the result easier to inspect.
Large effects do not imply a unique circuit
Transformers can contain redundant or interacting paths. If patching component A restores a behavior and patching component B also restores it, those observations do not prove that both components are independently required. Either intervention may supply information that downstream computation can exploit.
The reverse also matters. A component can participate in the normal computation yet show a small single-site patch effect because related information remains available through another path. Single-site interventions measure the effect of that replacement in the surrounding intact network, not an abstract amount of responsibility.
Interactions can be tested with multi-site or path-specific interventions, but the number of possible combinations grows quickly. Exhaustively patching every neuron, head, token position, and layer also requires many forward passes. Coarser localization followed by targeted tests often produces a more tractable experiment than treating every internal element as an independent candidate.
Corruption choice can create misleading evidence
A perturbation should change the behavior being studied without introducing unrelated disruption that dominates the model state. Replacing a token with an arbitrary token, adding noise to embeddings, or changing sentence structure can each produce different activation differences.
No corruption method is neutral. Noise has a scale and distribution. Token replacement changes lexical content. Prompt edits can alter tokenization and position. These details determine what the reference activation restores.
A useful robustness check repeats the intervention across multiple matched input pairs and, when appropriate, more than one plausible perturbation. A component whose effect appears only under one unusual corruption may still be interesting, but the evidence supports a narrower claim.
Patching supports mechanism tests, not semantic labels
Activation patching is strongest when attached to a concrete hypothesis: a state at a specified location contributes to a defined output difference for a controlled set of inputs. The intervention can then support or weaken that hypothesis through an observable output change.
It is weaker as a shortcut from activation to semantic label. A patched state is a high-dimensional object that can carry many signals at once, and downstream layers determine how those signals are used. Restoring an output does not establish that the patched vector contains one clean concept or that the same component serves the same role across unrelated prompts.
For developers inspecting transformer behavior, this distinction keeps the method useful. Patching can move an investigation from correlation toward intervention evidence, while the input pair, patch site, output metric, and surrounding network still define the scope of the result.