Beam search can return a shorter sequence even when a longer candidate contains locally plausible tokens at every position. The behavior follows directly from sequence scoring: autoregressive models multiply conditional token probabilities, or equivalently add their log probabilities. Since token probabilities are at most one, each additional token contributes a non-positive log term.
That arithmetic makes sequence length part of decoding. Beam width changes which candidates survive, but it does not remove the scoring effect. A decoder therefore needs a deliberate policy for comparing hypotheses of different lengths and for deciding when a completed hypothesis is good enough to stop the search.
Raw sequence scores accumulate with every token
For an input x and output sequence y = (y_1, ..., y_T), an autoregressive model factors the conditional probability as
p(y | x) = product_t p(y_t | y_<t, x)Decoders normally work in log space:
S(y) = sum_t log p(y_t | y_<t, x)Every term in that sum is zero or negative. Extending a hypothesis can therefore keep its raw log score unchanged only in the limiting case of probability one; normally the score decreases.
Consider two completed candidates:
A: 3 tokens, log score = -1.8
B: 6 tokens, log score = -2.4Under raw log-probability ranking, A wins because -1.8 > -2.4. That comparison is internally consistent with the model distribution. It does not establish that A is more complete, more informative, or more suitable for an application-level metric.
The distinction matters because beam search is a search procedure over model scores. It cannot repair a scoring objective merely by exploring more candidates.
Beam width exposes the scoring objective
At each decoding position, beam search expands the active hypotheses and retains a limited number according to their scores. A wider beam reduces pruning pressure within that approximation: more alternatives remain available for later expansion.
That can expose high-scoring completed sequences that a narrow beam would have pruned. If the model assigns unusually high total probability to short outputs, wider search can make those outputs easier to find. In that case, lower task quality with a wider beam is not evidence that wider search is intrinsically defective. The search can be doing a better job of optimizing a model score that is misaligned with the desired output properties.
This separation is useful during debugging. A search error means the decoder failed to recover a candidate that scores better under its stated objective. A model or scoring error means the recovered high-scoring candidate is undesirable according to the task criterion. Changing beam width mainly addresses the first category.
Length normalization changes candidate ordering
A common response is to normalize accumulated log probability by a function of sequence length. A simple form is
S_norm(y) = S(y) / T^alphawhere T is the sequence length and alpha controls the strength of normalization.
With alpha = 0, this reduces to the raw score. Positive values reduce the magnitude of the accumulated negative score for longer sequences. The resulting ranking is not the same probability ranking as the original model; it is a modified decoding objective.
For the earlier example and alpha = 1:
A: -1.8 / 3 = -0.60
B: -2.4 / 6 = -0.40Candidate B now ranks above A. Nothing about the underlying token probabilities changed. Only the rule used to compare complete sequences changed.
This is a central implementation boundary. Length normalization is not a neutral numerical correction. Its formula and coefficient encode a preference about how accumulated evidence should scale with output length.
A length reward expresses a different policy
Another scoring family adds a reward per generated token, often with a cap or another bound:
S_reward(y) = S(y) + lambda * min(T, L)Here lambda is the reward magnitude and L limits how far the reward applies. The raw model score still decreases as tokens are appended, while the reward offsets part of that decrease over the permitted range.
Normalization and additive rewards can produce different rankings because they transform scores differently. Dividing by a length function changes the scale of the whole accumulated score. Adding a reward changes the incremental preference associated with extending a sequence. They should not be treated as interchangeable parameters with different syntax.
The suitable choice depends on the output contract. Translation, transcription, constrained generation, and open-ended text generation can have different acceptable length distributions. A coefficient calibrated for one distribution can distort another.
End-of-sequence probability is part of the mechanism
Length behavior is also tied to the end-of-sequence token. A hypothesis becomes complete when the decoder emits the designated termination token. The probability assigned to that token competes with probabilities for ordinary continuation tokens at every eligible position.
A model that assigns substantial termination probability early can create short complete candidates with competitive raw scores. Search then determines whether those candidates remain visible; the model supplies the probabilities that make them competitive.
Minimum-length constraints can block termination before a chosen position. Such a constraint is useful when the application has a genuine lower bound, but it changes the allowed output space. It does not recalibrate the model’s probability estimates. Once the minimum is reached, the same score relationships can reappear.
Maximum length has a similar boundary role. It prevents unbounded decoding and provides a systems limit, but it should not be confused with a scoring correction. A decoder forced to stop at a cap may terminate without the model having selected its normal end token, depending on the implementation.
Stopping criteria must match the score transformation
Beam search maintains partial hypotheses and completed hypotheses at the same time. Stopping as soon as one beam finishes is generally different from stopping when no active hypothesis can surpass the best completed candidate under the decoder’s scoring rule.
For raw log probability, extending a partial hypothesis cannot increase its accumulated log score. That monotonic property can support upper bounds for unfinished candidates. Once the best unfinished score cannot beat the best finished score, further expansion cannot improve the result under that raw objective.
A modified length score complicates the bound. Normalization or a positive length reward can improve a candidate’s transformed score after extension even while its raw log probability decreases. A stopping rule copied from raw-score decoding can therefore terminate too early if it ignores the transformation.
This is one reason scoring and stopping should be designed together. The decoder needs an upper bound that is valid for the actual ranking function, not merely for the model’s unmodified log probability.
Compare length behavior on the target distribution
A single average output length is too coarse to diagnose beam behavior. Two systems can have the same mean while differing substantially across short and long inputs. Evaluation is more informative when candidate length is examined alongside the task signal that matters.
For paired input-output tasks, useful checks include the output-to-reference length ratio across input-length buckets, the frequency of termination near configured minimum or maximum bounds, and task quality across the same buckets. For applications without a single reference length, explicit product constraints or structured completion checks can provide a more relevant signal than a corpus-wide target length.
Beam width should also be varied during evaluation. If increasing the beam consistently shifts outputs toward a length extreme while improving the decoder’s internal score, that pattern points toward the scoring objective or model distribution rather than simple search insufficiency.
Length parameters should be evaluated on held-out data representative of the intended workload. Tuning them against the final test set folds test information into decoder selection and weakens the value of that test as an independent estimate.
Keep model probability and decoder preference separate
Raw sequence probability, normalized score, additive length reward, minimum-length constraints, and stopping rules answer different questions. Combining them into one opaque score field makes failures harder to interpret.
A practical decoder can retain the raw accumulated log probability separately from any adjusted ranking score. That separation makes it possible to inspect whether a candidate won because the model preferred it or because the decoding policy altered its rank. It also makes changes to length coefficients auditable without pretending that model probabilities changed.
Beam search length bias is therefore less about one universal correction than about preserving a clean boundary between probability estimation and decision policy. Once a decoder changes cross-length ranking, that transformation becomes part of the application objective and its stopping logic must respect the same definition.