Beam search keeps several partial sequences alive while decoding, but the score used to compare those sequences can create a systematic preference for particular lengths. With the common sum of token log probabilities, each additional token contributes a value at or below zero. A completed sequence can therefore lose score simply by continuing, even when the continuation is plausible.
This is not only a property of beam width. It comes from the objective used to rank hypotheses. Changing the beam size changes how much of the search space is explored; changing the scoring rule changes which sequences the search considers preferable.
Raw sequence probability accumulates a length cost
For an autoregressive model, the conditional probability of a sequence y = (y_1, ..., y_T) given input x factors as:
P(y | x) = product_t P(y_t | y_<t, x)Beam implementations normally work in log space:
S_raw(y) = sum_t log P(y_t | y_<t, x)Every conditional probability is at most one, so every added log-probability term is non-positive. Extending a hypothesis cannot increase its raw log-probability score.
Consider two completed candidates:
A: -0.30 -0.25 -0.40 = -0.95
B: -0.20 -0.20 -0.20 -0.20 -0.20 = -1.00Candidate A wins under the raw score even though B has a less negative average token score. The comparison is valid for the model probability as defined, but it may not match the application’s desired sequence-length behavior.
The end-of-sequence token participates in this process. Once a hypothesis emits that token, it becomes a completed candidate and no longer accumulates token costs. If the model gives substantial probability to ending early, raw scoring can make short completions especially competitive.
Beam width and score design solve different problems
A narrow beam can prune a prefix that later would have produced a high-scoring completion. Increasing beam width reduces that particular search limitation by retaining more alternatives.
It does not remove the length preference encoded by raw sequence probability. A wider beam can expose that preference more clearly because it has a better chance of retaining short paths that the model scores highly.
This distinction matters when diagnosing output that becomes shorter as beam width grows. Treating the symptom only as insufficient search can lead in the wrong direction. Search quality is defined relative to the scoring objective. A decoder can search that objective more thoroughly and still produce sequences that are undesirable for the application.
The useful separation is:
search procedure -> which hypotheses remain available
scoring rule -> which available hypothesis is preferredBoth affect the returned sequence, but they are different controls.
Length normalization changes the ranking objective
A common adjustment divides accumulated log probability by a function of sequence length. The simplest form uses the token count directly:
S_norm(y) = S_raw(y) / TThis score is the average log probability per token. It no longer ranks completed sequences by their model probability. Instead, it favors sequences whose tokens have high average conditional probability.
That change can reverse the earlier example:
A: -0.95 / 3 = -0.317
B: -1.00 / 5 = -0.200Under average log probability, B ranks above A.
More general schemes use a tunable exponent or another monotonic function:
S_alpha(y) = S_raw(y) / length(y)^alphaThe parameter controls how strongly length affects the transformed score. alpha = 0 recovers the raw sum. Other values define different ranking behavior rather than correcting the raw probability into a universally valid score.
That point is easy to miss. Length normalization is a decoding objective chosen for a task. It is not an identity that preserves the original sequence distribution.
A token reward is a different transformation
Another approach adds a constant reward for each generated token:
S_reward(y) = S_raw(y) + r * TFor positive r, longer candidates receive additional score. This can offset some of the accumulated negative log probability.
The reward has a different shape from division by length. With normalization, the effect depends on both total score and length. With an additive reward, extending a candidate changes the score by the new token log probability plus r.
Suppose a prefix has score -2.0, and the next token contributes -0.4. With r = 0.3, the effective change for that extension is -0.1. A token with log probability below -0.3 still lowers the transformed score; one above that threshold raises it.
This makes the reward parameter interpretable as a per-token offset, but its suitable value depends on the model, tokenization, and target output distribution. A reward calibrated for one tokenizer cannot be assumed to transfer unchanged to another because sequence lengths can differ for the same rendered text.
Partial and completed hypotheses need consistent treatment
Beam search compares candidates before all of them have reached the same length. A score transformation that is sensible for final reranking may behave differently when used for pruning partial hypotheses.
For example, average log probability can give a short prefix a strong score based on only a few easy tokens. A longer prefix may have accumulated evidence that becomes useful later but rank lower at the current decoding point. If pruning uses the transformed score directly, the normalization affects search trajectories as well as final selection.
Some systems therefore distinguish the score used to maintain the active beam from the score used to rank completed candidates. That design can preserve a search policy based on accumulated model score while applying a separate length-aware criterion at completion.
The separation introduces another requirement: stopping logic must agree with the scoring scheme. Under raw log probability, extending a prefix cannot improve its score, which provides a useful upper-bound property. A positive token reward or some normalized objectives can break that simple bound. A decoder that stops as soon as one completed sequence looks strongest under raw-score assumptions may terminate too early after the scoring rule changes.
Token count is not text length
Length-aware scoring usually operates on model tokens because decoding itself advances token by token. Token count is not equivalent to characters, words, bytes, or semantic content.
Two strings with similar visible length can require different token counts. The difference can depend on language, whitespace, punctuation, vocabulary construction, and tokenizer rules. A score penalty or reward tied to tokens therefore interacts with tokenization.
This becomes relevant when an application has an external length constraint. If the product requirement concerns characters, words, JSON fields, or another structural unit, token-level normalization is only an indirect control. Explicit constrained decoding or post-generation validation may match that requirement more closely.
The end-of-sequence token also needs a defined convention. Implementations can differ on whether the terminal token contributes to the length term. Either convention can be used, but evaluation and inference must use the same definition if score values are compared.
Score tuning belongs with task-level evaluation
A length parameter should be evaluated against the behavior the application actually needs. Looking only at model log probability can favor the unmodified model objective by construction, while looking only at average output length can hide quality regressions within that average.
A useful evaluation records output-length distribution alongside task metrics and failure categories. For structured generation, that may include completion validity and truncation. For translation or summarization, it may include adequacy-sensitive metrics plus length ratios relative to references. The exact measures depend on the task; the scoring parameter has no task-independent optimum.
Beam width should be varied separately during evaluation. If a length adjustment works only at one narrow beam size, the result may reflect an interaction between pruning and scoring rather than a stable ranking preference.
The central implementation boundary is simple: beam search explores candidates, while its score defines preference among them. Length-aware scoring is most predictable when that preference is explicit, its token-count convention is fixed, and stopping logic is checked against the transformed objective.