Beam search is a common way to decode sequence models when choosing the most likely token at every step is too shortsighted. It keeps several partial candidates alive, expands them, and repeatedly retains the strongest alternatives.

There is a subtle problem: the score used for a sequence usually accumulates one log-probability per generated token. Because token probabilities are at most 1, their log-probabilities are normally non-positive. Extending a sequence therefore tends to make its raw cumulative score smaller. When finished candidates of different lengths compete directly, this can create a preference for outputs that end too early.

Length normalization changes the ranking so that longer candidates are not penalized merely for containing more token decisions. This article builds the scoring mental model, works through a small example, and shows how to use length-aware scoring without assuming that longer output is automatically better.

Start with the score of a generated sequence

Consider an autoregressive model generating tokens y1, y2, ..., yT from an input x. The model factorizes the sequence probability as:

P(y1, ..., yT | x)
  = P(y1 | x)
  * P(y2 | y1, x)
  * ...
  * P(yT | y1, ..., yT-1, x)

Implementations normally add log-probabilities instead of multiplying probabilities:

log P(y | x) = sum_t log P(yt | y<t, x)

The two forms rank complete sequences identically before any extra decoding penalty is introduced. Log space is also numerically convenient because products of many small probabilities become sums.

Suppose two finished candidates have these simplified token probabilities, including an end-of-sequence token:

candidate A: 0.70, 0.60
candidate B: 0.80, 0.80, 0.80, 0.80

Their raw probabilities are:

A: 0.70 * 0.60                   = 0.42
B: 0.80 * 0.80 * 0.80 * 0.80   = 0.4096

Even though every local decision in B has probability 0.80, the product over four decisions is slightly below A’s product over two. In log space the same effect appears as additional negative terms in the sum.

This does not prove that a probabilistic model is mathematically wrong. The probability of a complete sequence really is the product of its conditional probabilities. The practical issue is different: the highest-probability complete sequence under the model may not have the length or task quality that an application wants.

Understand what beam search is actually comparing

With beam width k, beam search retains up to k promising partial hypotheses at each decoding step. A simplified loop looks like this:

beam = [empty sequence]

repeat:
    expand each unfinished hypothesis with candidate next tokens
    score the expanded hypotheses
    keep the strongest k candidates
    stop according to the decoder's termination rule

A beam is not a guarantee that the globally highest-scoring sequence will be found. Search remains approximate because candidates that fall out of the beam cannot recover later.

Length handling adds another complication. At a given step, unfinished candidates may have equal generated length, while finished candidates can have different lengths. A decoder therefore needs a clear policy for:

  • how partial hypotheses are ranked during search;
  • how completed hypotheses are ranked against one another;
  • whether an end-of-sequence token contributes to length;
  • when search can safely stop.

These details are implementation choices, not universal properties of beam search. When reproducing a system, inspect its exact scoring and stopping definitions rather than relying only on a parameter name such as length_penalty.

Normalize the score instead of rewarding length blindly

A simple length-normalized score divides cumulative log-probability by a function of sequence length:

score(y) = log P(y | x) / L(y)^alpha

where:

  • L(y) is the length used by the decoder;
  • alpha controls the strength of normalization;
  • alpha = 0 recovers the raw log-probability score.

For a teaching example, suppose:

A: length = 2, log-probability = -0.87
B: length = 4, log-probability = -0.89

Without normalization, A wins because -0.87 > -0.89.

With alpha = 1:

A: -0.87 / 2 = -0.435
B: -0.89 / 4 = -0.2225

Now B wins because its average log-probability per length unit is higher.

This example exposes an important point: normalization does not discover which candidate is semantically better. It changes the decoding objective. The system designer is saying that raw sequence probability alone is not the desired ranking criterion.

Different libraries and papers use different length functions. Some use a plain power of token count; others use shifted or scaled formulas. The meaning of a particular alpha therefore depends on the formula around it.

Separate three different length problems

Developers often describe every short-output problem as “beam search length bias,” but several causes can produce similar symptoms.

The model may genuinely prefer ending

The model assigns probability to the end-of-sequence token just like other output choices. If it gives that token too much probability after a short prefix, the decoder may terminate early even before length normalization becomes the dominant issue.

That can come from training data, model quality, fine-tuning, or a mismatch between training and deployment inputs. A decoding penalty can change the symptom without fixing the learned distribution.

Raw cumulative scoring can disadvantage extensions

Each additional token contributes another non-positive log-probability. When sequences of different lengths are compared by their sums, longer candidates need sufficiently strong conditional probabilities to overcome the accumulated score.

Length normalization directly targets this ranking effect.

Search can prune a good continuation too early

A future high-quality sequence can disappear if its partial hypothesis falls outside the beam. Increasing beam width may preserve more alternatives, but it also increases decoding work and does not guarantee better task-level quality.

These three cases need different responses. Treating them as one knob-tuning problem makes debugging unnecessarily difficult.

Use a minimum length only for hard task constraints

A decoder can often forbid the end-of-sequence token until a minimum number of tokens has been generated. This is useful when outputs below a known length are structurally invalid.

For example, imagine a task whose output format requires three fields:

priority | owner | action

If the tokenizer and grammar make it impossible to produce a valid record below some known boundary, a minimum-length constraint may be defensible.

It is much weaker as a general cure for poor summaries or translations. If a model wants to finish at token 8 and you forbid termination until token 20, it may generate filler or low-probability continuations rather than a better answer.

Use a hard minimum to encode a hard requirement. Use evaluation and model improvements for quality problems.

Do not confuse normalization with a length reward

Another possible scoring rule adds an explicit reward per generated token:

score(y) = log P(y | x) + beta * L(y)

For positive beta, each extra length unit receives a fixed bonus. This is not the same transformation as dividing by a length-dependent term.

A reward can be useful in systems where insertion and deletion behavior needs direct control, but it has a distinct failure mode: if the bonus is too strong, the decoder can prefer unnecessarily long outputs simply because more tokens earn more reward.

Length normalization can also overcorrect, especially with an aggressive exponent. The general lesson is to reason from the exact equation. Names such as “length penalty,” “length bonus,” and “normalization” are not precise enough to tell you what a decoder does.

Tune against task quality, not output length alone

A useful tuning experiment records both quality and length statistics on a representative validation set.

For each candidate normalization setting, measure at least:

task metric
output length distribution
fraction of suspiciously short outputs
fraction of suspiciously long outputs
latency or generated-token cost

The task metric depends on the application. A structured generation task may use exact match or field-level correctness. A translation or summarization system may need automated metrics plus human review. An application with downstream execution should evaluate whether the generated result actually succeeds at that downstream task.

Do not select a normalization strength because its mean output length matches a reference mean. Two systems can have the same average length while making very different mistakes.

Look at the distribution and inspect examples near the tails. If short failures disappear but verbose, repetitive outputs increase, the normalization is trading one error mode for another.

Keep scoring consistent with stopping logic

Stopping beam search is easy when every candidate uses a raw cumulative score with predictable monotonic behavior. Length-dependent transformations can make safe early stopping more subtle because a partial hypothesis’s eventual normalized score depends on its future length and future token probabilities.

A production decoder should use the stopping rule defined for its scoring implementation rather than a hand-written shortcut such as:

stop as soon as the current best hypothesis ends

That shortcut can be wrong because another live hypothesis may later produce a better completed sequence.

Similarly, do not assume that the first completed candidate is the final winner. Many implementations maintain completed hypotheses separately and continue until a criterion shows that further expansion is unnecessary or a configured limit is reached.

If you implement beam search yourself, test the scorer and termination rule together with tiny synthetic distributions where you can enumerate every possible sequence. Exhaustive enumeration is impractical for real models but excellent for validating decoder logic on a miniature vocabulary and short maximum length.

Remember that token length is not human length

Length penalties normally operate on generated tokens because tokens are what the model scores. Tokens are not equivalent to words, characters, sentences, or semantic content.

A four-token output in one language or tokenizer can represent a different amount of text from four tokens in another. Code, numbers, punctuation, and uncommon words can also split differently.

This matters when the product requirement is expressed in human units such as “roughly three sentences” or “under 100 words.” Token-level normalization can influence generation, but it is not a precise enforcement mechanism for those requirements.

If a hard external limit exists, validate the final output in the unit that actually matters. If a soft stylistic target exists, evaluate that target directly rather than inferring it from token count.

Account for beam width and compute cost

Length normalization and beam width affect different parts of decoding.

The normalization rule changes how candidates are ranked. Beam width changes how many alternatives survive each pruning step. A wider beam can uncover candidates that a narrow beam discarded, but it requires more scoring and candidate management and can increase memory use and latency depending on the implementation.

It is therefore possible for the best normalization strength at one beam width to behave differently at another. Treat decoding configuration as a system rather than tuning each parameter in isolation.

For interactive language generation, beam search may not be appropriate at all. Greedy decoding can be sufficient for constrained deterministic tasks, while sampling methods are often used when output diversity matters. Length normalization solves a specific ranking issue inside sequence search; it is not a reason to introduce beam search where the application does not need it.

Diagnose failures before changing the penalty

When outputs are unexpectedly short, inspect concrete examples and ask questions in this order:

  1. Did the model assign high probability to ending? If so, the learned distribution may be the main issue.
  2. Did a promising longer hypothesis exist but lose because of cumulative score? Length-aware ranking may help.
  3. Was the promising hypothesis pruned before it could become strong? Beam width or search strategy may matter.
  4. Did the decoder stop before all competitive hypotheses were resolved? Check termination logic.
  5. Is the desired length actually a product constraint? Encode and evaluate that requirement explicitly.

Logging a few top hypotheses with their cumulative scores, normalized scores, lengths, and termination states is often more informative than changing alpha repeatedly and observing only final text.

For large models, full token-level traces can be expensive to retain in production. A sampled diagnostic mode or offline reproduction on representative requests is usually enough to reveal the ranking behavior.

Know when a simpler approach is better

Length normalization is useful when all of these are approximately true:

  • you are using beam-style sequence search;
  • finished candidates of different lengths compete;
  • raw cumulative scoring produces a measurable short-output problem;
  • task-level evaluation shows that a length-aware ranking improves useful output.

It is less compelling when output length is fixed, when a grammar determines valid termination, or when greedy decoding already meets the application’s quality requirements. It is also the wrong first fix when the model fundamentally does not know the requested content.

A decoder can only rank continuations the model makes available. Changing the ranking cannot manufacture missing knowledge, repair bad training labels, or guarantee factual correctness.

Conclusion

Beam search ranks sequences by scores accumulated across token decisions. Raw log-probability has a natural length effect because every extension adds another non-positive term. Length normalization changes that objective so longer candidates can compete on a more length-aware basis.

The practical approach is to make the scoring equation explicit, distinguish model termination from search and ranking failures, and tune against task quality plus the full output-length distribution. Use hard length constraints only for genuine constraints, verify stopping logic with the chosen scorer, and remember that token count is only a proxy for the length users perceive.

Length normalization is most useful as a controlled correction to a diagnosed decoding problem, not as a generic setting that makes beam search better.

Artificial Intelligence Machine Learning Sequence Models Beam Search Inference