Beam search compares multiple partial outputs while autoregressive generation advances token by token. A common scoring rule adds token log probabilities along each candidate sequence. That rule is mathematically consistent with sequence probability, but it also creates a structural preference that developers can miss: extending a sequence normally makes its accumulated log score smaller.
This matters whenever candidates of different lengths compete. A decoder can rank a short completed sequence above a longer candidate even when the longer candidate is more useful for the application. Length normalization and length penalties modify that ranking, but they also change the objective being optimized.
Raw sequence scores accumulate negative terms
For a generated sequence y_1, ..., y_T, an autoregressive model assigns:
log p(y_1, ..., y_T | x)
= sum_t log p(y_t | y_<t, x)Token probabilities lie between zero and one, so their logarithms are non-positive. Adding another token therefore cannot increase the raw log probability of a sequence. If two candidates share the same prefix and one stops while the other continues, the continued candidate receives at least one additional non-positive term.
Consider two completed candidates:
candidate A token log scores: -0.2, -0.3
raw score: -0.5
candidate B token log scores: -0.2, -0.3, -0.2, -0.2
raw score: -0.9Under raw accumulated score, candidate A ranks above candidate B because -0.5 > -0.9. The comparison says that the complete short sequence has greater model probability. It does not say that the short sequence is preferable under an application’s quality criterion.
Beam search exposes this distinction because completed hypotheses can have different lengths.
The end token participates in the competition
Sequence length is not chosen outside the model. In ordinary autoregressive decoding, an end token competes with other next-token options. Once that token is selected, the hypothesis becomes complete.
A model that assigns substantial probability to ending early can therefore produce short candidates with strong raw sequence scores. Beam search may retain those candidates while longer hypotheses continue accumulating negative log scores.
Minimum-length constraints can block the end token before a chosen position, but that is a hard decoding constraint rather than score normalization. It changes which sequences are permitted. A length penalty instead changes how permitted sequences are ranked.
The distinction matters in implementation. A hard constraint can prevent an otherwise valid short answer from existing in the candidate set. A scoring adjustment keeps the candidate but changes its position relative to alternatives.
Length normalization changes the ranking objective
A simple normalization divides accumulated log score by sequence length:
normalized_score = log_probability / TFor the earlier candidates:
candidate A: -0.5 / 2 = -0.25
candidate B: -0.9 / 4 = -0.225Candidate B now ranks above candidate A. Nothing about the model probabilities changed. Only the decoder’s comparison rule changed.
This form resembles average token log probability. It reduces the direct accumulation effect, but it does not recover a universal notion of output quality. A candidate can obtain a strong average score by extending through locally predictable tokens, while another candidate may express the required content more compactly.
Many decoders use a parameterized length penalty instead of plain division. A generic form is:
adjusted_score = log_probability / penalty(T)The exact penalty(T) is implementation-specific. Some libraries use a power of length; others use a shifted length formula or expose a parameter with different semantics. A numeric penalty value cannot be transferred safely between decoding implementations without checking the scoring definition.
Beam pruning happens before final ranking
Length handling is often discussed as if it only sorts completed sequences at the end. Beam search also prunes partial hypotheses during generation.
At each decoding position, the beam retains a limited number of candidates. A hypothesis removed at an early position cannot return later, even if its eventual completed score would have been strong under the final ranking rule.
This creates an interaction between beam width, partial-score comparison, end-token handling, and length adjustment. Two decoders can use the same model and nominal beam width yet produce different outputs if they apply normalization at different stages or use different rules for completed hypotheses.
For this reason, a final scoring formula does not fully specify beam-search behavior. The pruning policy is part of the decoding algorithm.
Completed and active hypotheses need comparable treatment
A decoder may hold both completed sequences and active sequences that can still grow. Comparing them requires care because an active candidate has not yet paid the score cost of future tokens.
Suppose an active sequence currently has a strong adjusted score. Its final score can still move as tokens are appended and its length penalty changes. Treating its current adjusted score as directly equivalent to a completed candidate can produce premature stopping unless the stopping rule accounts for the range of scores the active candidate can still reach.
Libraries handle this with different bookkeeping and stopping criteria. Some keep completed hypotheses separately and stop once enough finished candidates satisfy an internal bound. Others continue until all beams finish or a maximum length is reached.
Application code should treat early stopping, beam width, and length penalty as coupled decoding settings rather than independent switches.
Score calibration does not survive arbitrary penalties
Raw sequence log probability has a direct probabilistic interpretation under the model. Once a decoder divides or rescales that score with a length-dependent function, the result is a ranking score, not generally the log probability of the generated sequence.
That distinction affects downstream systems. An adjusted beam score should not automatically be treated as a calibrated confidence value. Comparing such scores across requests with different decoding settings can be especially misleading because the ranking transformation itself has changed.
If a system needs model likelihood for analysis, retain the raw token or sequence log probabilities separately from the beam-ranking score. The decoder can use one quantity for search while observability code records the other.
Length settings should match the output contract
The useful setting depends on the output distribution the application expects. Short labels, bounded structured fields, summaries, and open-ended text have different acceptable length ranges. A penalty that helps one output contract can distort another.
Evaluation should therefore inspect length together with task quality. Aggregate output length alone cannot establish that decoding improved, and a quality metric alone can hide a systematic shift toward overly short or overly long responses.
It is also useful to separate model behavior from decoder behavior during diagnosis. If changing only the beam scoring rule causes a large length shift, the decoder is contributing materially to the observed output distribution. If the shift persists across scoring rules, token probabilities and end-token behavior deserve closer inspection.
Beam search does not merely search for a fluent continuation. Its scoring and pruning rules define which model-supported sequence wins under finite search. Length handling is part of that decision rule, so it belongs in the same configuration surface as beam width, stopping behavior, and token constraints.