Control Beam Search Length Bias with Sequence Scoring

Beam search can prefer a short completed sequence even when a longer continuation looks locally plausible at every token. The effect follows from the score being optimized. If a decoder ranks complete hypotheses by the sum of token log probabilities, every additional token contributes a value that is at most zero. Extending a sequence therefore cannot increase its raw accumulated log probability.

This property is not a defect in probability theory. A sequence probability is a product of conditional probabilities, and its logarithm is their sum. The implementation concern appears when raw sequence probability is also used as the ranking objective for outputs whose lengths vary.

Raw sequence scores contain a length effect

For an autoregressive model, a generated sequence y_1, ..., y_T conditioned on input x has log probability:

S_raw(y) = sum_t log p(y_t | y_<t, x)

Since each conditional probability lies in [0, 1], each log term is non-positive. Consider two completed hypotheses:

A: -0.20 + -0.25              = -0.45
B: -0.10 + -0.12 + -0.13 + -0.14 = -0.49

Under the raw sum, A ranks above B because -0.45 is greater than -0.49. Yet the average token log probability is lower for A than for B. The two scoring views answer different questions: total sequence probability versus probability accumulated per generated position.

The end-of-sequence token participates in this competition when the model uses one. A hypothesis that emits it early becomes complete and stops accumulating ordinary continuation terms. Whether that hypothesis survives depends on the beam implementation and scoring rule, but the raw-score arithmetic already favors avoiding extra negative terms.

Beam width does not remove the scoring preference

Increasing beam width keeps more partial hypotheses alive. It can reduce search errors caused by pruning a promising prefix too early, but it does not change the objective used to rank completed candidates.

If the final selection uses raw accumulated log probability, a wider beam can expose more high-scoring short completions rather than neutralize the length effect. Search capacity and sequence scoring are separate controls.

This distinction also separates beam search from greedy decoding. Greedy decoding selects one token at each position and never compares complete sequences of different lengths as a set. Beam search retains several prefixes, making the sequence-level score explicit in pruning and final selection.

Length adjustment changes the objective

A common response is to normalize or penalize scores as a function of generated length. A simple form divides the accumulated log probability by a positive length-dependent term:

S_adjusted(y) = S_raw(y) / f(T)

With f(T) = T, the score becomes mean token log probability. Other systems use a tunable exponent or another monotonic function. These choices are not algebraically equivalent to maximizing the model’s original sequence probability. They define a modified decoding objective that changes the relative ranking of lengths.

That change can be intentional. If the application expects outputs within a useful length range, raw probability may not align with the desired selection behavior. The relevant parameter then belongs to decoding policy, not to the model distribution itself.

A length adjustment can also overcompensate. Strong normalization may favor continuations that remain moderately probable for many positions, even when a concise completion is appropriate. There is no length rule that is neutral across all tasks because the desired output length depends on the data and output contract.

Partial and completed hypotheses need compatible handling

Beam implementations differ in how they retain finished hypotheses and compare them with unfinished prefixes. Some keep completed candidates in a separate set while the active beam continues. Others allow finished entries to occupy beam capacity. Early stopping rules can terminate search once further active hypotheses cannot produce a preferred completion under the decoder’s scoring assumptions.

These details matter because length adjustment may be applied at different times. Applying a final normalization only after search is not generally equivalent to using an adjusted score during pruning. A prefix discarded under raw scoring cannot reappear later even if its eventual normalized completion would have ranked highly.

For the same reason, comparing two beam-search implementations requires more than matching beam width. Sequence score definition, end-of-sequence handling, minimum and maximum generation lengths, finished-hypothesis retention, and stopping criteria can all alter the candidate set.

Score diagnostics should retain both views

When output length shifts after a decoding change, storing only the selected text hides the mechanism. Evaluation can retain raw accumulated log probability, generated token count, adjusted score, and completion status for each final candidate.

Those values make ranking changes inspectable. If a candidate wins only after normalization, the decoder has deliberately traded total sequence probability for its length-adjusted objective. If all candidates hit a maximum token limit, the observed length distribution says more about the boundary than about natural completion behavior.

Aggregate task metrics still matter because a preferred score does not establish output quality. For structured generation, the useful signal may be parse success or constraint satisfaction. For translation or summarization, evaluation may depend on reference-based or human-defined criteria. Length statistics are diagnostic context rather than a substitute for task evaluation.

Length controls operate at different layers

A minimum generation length can prevent an end-of-sequence token from being accepted too early. A maximum length truncates the search space. A length-adjusted sequence score changes ranking among hypotheses. These controls can produce similar visible changes in output length while acting through different mechanisms.

Treating them as interchangeable makes decoder behavior harder to reason about. A minimum length imposes a hard constraint; normalization changes preferences inside the permitted space. A maximum length creates a hard boundary that can force unfinished outputs. The appropriate control follows from the intended output contract.

Beam search exposes a useful separation between model probability and decoding preference. Once hypotheses of different lengths compete, the exact sequence score becomes part of application behavior. Recording that score definition alongside beam width and stopping rules keeps output-length changes attributable to an explicit inference policy rather than an opaque property of the model.