A transformer can expose a clear internal pattern without that pattern being responsible for the output under inspection. Activation patching addresses this gap by changing an internal state and measuring the downstream effect. Instead of asking whether a feature is visible at a layer, it asks whether replacing a selected state changes a defined model behavior.
The method is simple in form but sensitive to experimental design. A patch has meaning only relative to the paired inputs, the patched location, the replacement value, and the output metric. Changing any of those can change the causal question being tested.
A patch connects two forward passes
Activation patching commonly starts with two inputs. A clean input produces the behavior of interest. A corrupted input alters a controlled part of the input so that the behavior weakens or changes.
During the clean forward pass, the implementation records an activation at a chosen internal site. During the corrupted pass, it replaces the corresponding activation with the recorded clean value and continues the remaining computation.
A compact representation is:
clean input -> clean activation a_c -> clean output
corrupted input -> activation a_x -> corrupted output
corrupted input -> replace a_x with a_c -> patched outputIf the patched output moves toward the clean output, the selected internal state carries causal influence for that metric under this intervention. The result does not establish that the site is the only route carrying the relevant information. Other components can act redundantly or interact with the patched state.
This distinguishes patching from a passive probe. A linear readout, token projection, or visualization can show information encoded in an activation. Patching tests the consequence of substituting that activation while the rest of the corrupted computation remains in place.
The metric defines the behavior being restored
A patch does not produce a generic measure of causal relevance. It produces a change in a chosen output statistic.
For a next-token comparison between target token t and alternative token r, a useful metric can be the logit difference:
m(x) = logit_t(x) - logit_r(x)Let m_clean, m_corrupt, and m_patch denote this value for the three runs. The raw patch effect is:
effect = m_patch - m_corruptThis quantity reports how much the intervention moves the corrupted run on the selected axis. It avoids treating the full vocabulary distribution as a single scalar notion of restoration.
A normalized score is sometimes useful when clean and corrupted metrics differ substantially:
recovery = (m_patch - m_corrupt) / (m_clean - m_corrupt)That ratio requires care. It is undefined when the denominator is zero and unstable when the clean-corrupted gap is very small. Values outside the interval from zero to one are also possible because an intervention can overshoot the clean metric or move in the opposite direction. The ratio should therefore be treated as a derived experimental statistic, not as an intrinsic percentage of information stored at a site.
Patch granularity changes the causal claim
A transformer activation has several axes, so the phrase “patch a layer” can hide materially different interventions. A developer might replace the entire residual stream at one layer, one token position in that stream, the output of a single attention head, an attention pattern, or the output of a feed-forward block.
Replacing a full residual state is a broad intervention. It can restore information contributed by many earlier components at once. A strong effect at that location says that the substituted state matters downstream, but it does not isolate the component that originally wrote the relevant information.
Patching one attention-head output is narrower. Even then, the intervention tests the head output in the context of the selected paired runs. It does not establish that the head has one fixed semantic role across arbitrary prompts.
Token position matters for the same reason. Replacing a vector at the final prompt position tests a different computational path from replacing a vector attached to an earlier source token. A patching map over layers and positions is meaningful only when each cell corresponds to a clearly specified tensor slice.
Clean and corrupted inputs need controlled semantics
The corrupted input sets the baseline computation into which clean state is inserted. If corruption changes many unrelated properties at once, a successful patch can be hard to interpret.
Suppose an experiment tests whether a model tracks a named entity across a sentence. Replacing the entity with another token may also alter token count, subword segmentation, syntax, or position alignment. A patch at position i then might connect states that do not represent comparable input roles.
Tokenization should be checked before treating positions as aligned. Two visible strings can map to different token counts. When sequence lengths differ, a positional patch may require a more explicit correspondence than equal tensor indices.
The corruption also needs to change the target metric enough to create a useful contrast. If clean and corrupted runs already produce nearly identical outputs, restoration is difficult to distinguish from ordinary variation in the metric.
Replacement can create states outside the normal computation
Activation patching forms a hybrid forward pass. Earlier computation comes from the corrupted input, while a selected state comes from the clean input. The resulting combination need not be a state the model would naturally produce for either input.
This matters most when the patched activation interacts strongly with surrounding states. Attention and feed-forward computation are nonlinear across a full block, and later components can respond differently to a substituted vector than they do in the clean run. A large output shift demonstrates sensitivity to the intervention; it does not guarantee that the patched state acts independently of its corrupted context.
The same boundary applies to negative results. A site can participate in the clean computation yet show little restoration when patched alone. Relevant information may be distributed across several sites, encoded in a form that depends on other clean states, or recoverable through redundant paths left intact by the corruption.
Direction of replacement is part of the experiment
Clean-to-corrupted patching asks whether clean internal state can restore a behavior in a corrupted context. The reverse intervention asks a different question: whether corrupted state can damage the clean behavior when inserted into a clean context.
These effects need not be symmetric. Nonlinear downstream computation and interactions among components can make one replacement direction large and the other small. Reporting the direction explicitly prevents a causal effect from being mistaken for a property of the activation in isolation.
Ablation is different again. Replacing a state with zeros, a mean activation, or another baseline tests dependence on that baseline intervention. Patching from a matched run preserves a concrete alternative state generated by the same model, which can make the intervention easier to relate to a specific input contrast.
Sweeps reveal locations, not complete circuits
Running the same intervention across many layers, positions, or components produces a patching map. Such a map can identify locations where substitution has a large effect on the chosen metric. It is often a useful filter for more focused causal analysis.
The map should not be read as a complete circuit diagram. Patching one site at a time can miss interactions that appear only when several states are changed together. It can also assign strong effects to downstream locations that merely carry information produced elsewhere.
A late residual-stream patch, for example, may restore the target output because it imports the accumulated result of several earlier operations. That makes the location causally effective as an intervention point, but it does not show where the computation originated.
Component-level patches can narrow the search. Follow-up interventions can test candidate interactions, compare replacement directions, or patch groups of components selected by an explicit hypothesis. The evidence becomes stronger when each intervention rules out a concrete alternative account rather than merely adding another heat map.
Implementation hooks must preserve tensor semantics
Framework hooks make activation replacement convenient, but the tensor being intercepted must match the intended conceptual site. Transformer libraries differ in block structure, normalization placement, residual connections, fused projections, and returned intermediate values.
A hook named after an attention module might expose its projected output, an input tuple, or a value before a residual addition depending on the implementation. The experiment should identify the exact tensor shape and point in the forward graph before interpreting a patch as an attention-head or residual-stream intervention.
Inference settings also need consistency across paired runs. Dropout should not introduce unrelated stochastic differences. Tokenization, padding, attention masks, position identifiers, and generation state should be aligned with the experimental contrast. For autoregressive generation, patching cached states across different token histories requires extra care because the cache encodes prior context at specific positions.
Recording only the activation required for the patch reduces accidental ambiguity. A small experiment with explicit tensor coordinates is easier to audit than a broad hook system whose captured values are not tied to named model operations.
Causal evidence remains local to the intervention
Activation patching can support a precise statement: under a specified pair of inputs, replacing a specified internal state changes a specified downstream metric by a measured amount. That is stronger than observing correlation inside the model, but narrower than claiming a universal function for the patched component.
General claims require repeated contrasts that vary surface form while preserving the mechanism under test. Even then, the evidence concerns the tested distribution and intervention scheme. A component may participate differently when context, token position, or competing features change.
The most useful patching result is therefore not the largest score. It is an intervention whose tensor location, input contrast, baseline, direction, and output metric make the causal claim unambiguous enough to test again under new conditions.