Attention Entropy Measures Concentration, Not Causal Importance

An attention head can place most of its probability mass on one position and still provide little evidence that this position controls the final model output. The entropy of its attention weights captures concentration, not causal influence.

That distinction matters when attention maps are inspected as diagnostics. Entropy can reveal whether a head spreads mass broadly or focuses it narrowly for a given query. It cannot, by itself, establish that the highest-weight token carries the feature responsible for a downstream prediction.

Entropy summarizes the weight distribution

For one query position, let an attention head produce normalized weights a_1, ..., a_n after masking and softmax, with

a_i >= 0
sum_i a_i = 1

The Shannon entropy of that distribution is

H(a) = -sum_i a_i log(a_i)

A distribution concentrated on one position has entropy near zero. A uniform distribution over n available positions has entropy log(n). Between those limits, entropy increases as probability mass becomes more evenly distributed.

This quantity is a property of the normalized weights. It does not inspect the value vectors, the output projection, residual stream, later layers, or final task output.

The log base changes only the unit. Natural logarithms give entropy in nats; base-2 logarithms give bits. Comparisons should use the same convention.

Sequence length changes the upper bound

Raw entropy is not directly comparable across attention distributions with different numbers of available positions. A query that can attend to four positions has a maximum entropy of log(4), while one that can attend to 4,000 positions has a much larger maximum.

A common normalization is

H_norm(a) = H(a) / log(n)

for n > 1, where n counts positions that are actually available after masking. This maps the uniform distribution to 1 and a one-hot distribution toward 0.

Normalization solves only the changing upper bound. It does not make two heads semantically comparable. Different heads can operate on different representations, and the same entropy can arise from very different weight patterns.

Padding and causal masks also matter. Including masked positions in n would distort the normalization because those positions are not candidates in the distribution.

Equal entropy does not imply equal attention structure

Entropy compresses an entire probability vector into one scalar. That compression discards positional identity.

Consider two distributions:

A = [0.8, 0.1, 0.1, 0.0]
B = [0.0, 0.1, 0.1, 0.8]

They have the same entropy because they contain the same probability values. Yet their dominant mass points to different positions.

Even distributions with different shapes can produce similar entropy values. An entropy trend can therefore indicate a change in concentration without identifying which positions gained or lost mass. Position-aware analysis still requires the weights themselves or another representation of their structure.

Attention weights do not include value magnitude

For a single head, the attention output before the output projection can be written as

o = sum_i a_i v_i

where v_i is the value vector at position i.

Entropy sees only a_i. It does not see the direction or magnitude of v_i. A position can receive a large attention weight while its value vector contributes little along a downstream-relevant direction. Conversely, several modest weights can combine value vectors into a substantial output.

The output projection and multi-head aggregation add another boundary. A sharply focused head may be attenuated or transformed after its weighted sum, while another head or the residual path carries information that dominates later computation.

For that reason, low attention entropy is accurately described as concentrated routing inside that attention operation. Calling it high feature importance adds a claim that entropy itself does not support.

Residual paths separate attention from total influence

Transformer blocks do not usually replace the residual stream with the attention result. The attention contribution is combined with a residual path, followed by additional transformations according to the architecture.

This means an output can remain stable even when an individual attention pattern changes. The residual stream can already contain information needed by later layers, or other components can compensate for the changed contribution.

The reverse can also occur: a modest-looking attention shift can matter if it changes a direction that a later computation is sensitive to. Neither case can be inferred from entropy alone.

This is an implementation boundary rather than a defect in the metric. Entropy was defined to summarize a probability distribution. It was not defined to measure end-to-end causal effect.

Causal claims require interventions or attribution with explicit assumptions

If the question concerns causal influence on a model output, the evaluation has to connect an internal change to that output. Possible analyses include controlled ablations, activation replacement, value-vector interventions, or other attribution methods with clearly stated assumptions.

For example, replacing one head’s output with a reference value and measuring the change in a target logit tests a different property from measuring that head’s attention entropy. The intervention changes computation and observes a downstream response. Entropy only describes the original weight distribution.

Interventions also need care. Removing a component can create an internal state that is unlike states seen during normal inference, and effects can depend on the chosen baseline or replacement. A causal analysis therefore needs its intervention semantics reported alongside its result.

Entropy remains useful as a diagnostic

The limitation does not make attention entropy uninformative. It is useful for questions that are actually about concentration.

Across requests, entropy can expose heads that consistently use narrow or broad distributions. Across token positions, it can show where concentration changes. During model comparison, it can summarize distributional shifts when masking, context length, architecture, or serving configuration changes.

Those uses become clearer when entropy is paired with the exact scope being measured: layer, head, query position, mask, sequence length, and aggregation rule. Averaging entropy across heads or tokens creates another summary layer and can hide heterogeneous behavior.

The boundary is precise: attention entropy reports the shape of normalized attention weights. Claims about token importance, information content, or causal control require evidence from quantities or interventions that connect those weights to the rest of the model computation.