Beam search ranks partial sequences by scores accumulated across decoding steps. When that score is the sum of token log-probabilities, every additional token contributes a value that is normally non-positive. A longer candidate therefore has more opportunities to reduce its raw score, even when its continuation is locally plausible.
That property is not a defect in probability theory. It follows from comparing complete sequences with different numbers of conditional factors. It becomes an implementation concern when a decoder is expected to produce useful completions rather than simply rank sequences by unmodified model probability.
Sequence probability becomes an additive score
For a generated sequence y_1, ..., y_T conditioned on input x, an autoregressive model factorizes its probability as
P(y | x) = product_t P(y_t | y_<t, x)Beam search commonly works in log space:
S(y) = sum_t log P(y_t | y_<t, x)The logarithm turns a product into a sum and avoids multiplying many small probabilities. Since each conditional probability lies between zero and one, its logarithm is at most zero. Extending a sequence with another ordinary token cannot increase this raw cumulative log-probability.
This creates a structural length effect. Consider two completed candidates with raw scores -4.0 and -5.0. Under raw sequence score, -4.0 ranks higher regardless of whether the first sequence has three generated tokens and the second has ten. The comparison is mathematically valid for the model distribution, but it embeds sequence length into the ranking.
The end-of-sequence token participates in the same factorization when the model uses one. A candidate that reaches that token early stops accumulating further token terms. Beam implementations must therefore define both the score used for active hypotheses and the score used to compare completed hypotheses.
Normalization changes the objective used for ranking
A decoder can compensate for the cumulative length effect by transforming the raw score. A simple form divides by generated length:
S_norm(y) = S(y) / TThis is the mean token log-probability. It does not represent the same ranking objective as raw sequence probability. A longer candidate with a lower raw score can outrank a shorter candidate if its average token score is higher.
For example:
candidate A: S = -4.0, T = 4, S_norm = -1.0
candidate B: S = -6.0, T = 10, S_norm = -0.6Raw scoring selects A because -4.0 > -6.0. Mean log-probability selects B because -0.6 > -1.0. No model logits changed. Only the sequence-level ranking rule changed.
Many decoders use a parameterized length penalty rather than direct division by T. The exact function is implementation-specific. A parameter named length_penalty does not establish a universal formula, sign convention, or point in the search procedure. Reproducing behavior requires the actual scoring equation used by the serving or generation stack.
Search-time and final ranking need not use the same score
Beam search cannot retain every possible continuation. At each step it keeps only a bounded set of hypotheses, so pruning decisions affect which sequences remain reachable. This makes the timing of length normalization significant.
One implementation can use normalized scores while pruning active beams. Another can retain beams using raw cumulative scores and apply a length adjustment only when ranking completed sequences. Those procedures can return different outputs even if they use the same model, beam width, and nominal penalty parameter.
The difference comes from irreversible pruning. Once a partial sequence leaves the beam, a later final-score adjustment cannot restore its descendants. A scoring rule applied only at completion changes the ordering of surviving candidates; a scoring rule applied during expansion can also change which candidates survive.
This distinction matters when comparing libraries or moving a workload between inference systems. Matching the final penalty formula is insufficient if beam pruning, end-of-sequence handling, or completion bookkeeping differs.
Beam width does not remove the length effect
Increasing beam width keeps more alternatives alive, but it does not redefine the score. With raw cumulative log-probability, the same length-dependent ordering remains. A wider beam can reduce some search errors caused by early pruning, yet it cannot turn a raw sequence-probability objective into a normalized one.
Beam width also does not guarantee recovery of the globally highest-scoring completed sequence unless the search procedure and stopping rule provide the conditions needed for that guarantee. Practical decoders often include finite beam width, early stopping, minimum or maximum length constraints, token filters, and other processors. Each can alter the set of sequences that is actually considered.
The useful separation is between model scoring and search policy. The model supplies conditional token probabilities. Beam width controls retained hypotheses. Length normalization transforms sequence scores. Stopping rules decide when expansion ends. Treating these as separate mechanisms makes output differences easier to attribute.
Completion rules interact with normalized scores
A decoder also needs a rule for deciding when enough completed hypotheses exist. With raw scores, an unfinished hypothesis has a cumulative score that can only stay the same or decrease as tokens are appended. That property can support bounds for deciding whether continued expansion can still beat a completed candidate.
Length-normalized scores complicate such reasoning. The denominator changes as a hypothesis grows, so the final normalized score is not determined by the current raw score alone. Depending on the normalization function, extending a sequence can improve its normalized value even while its raw log-probability decreases.
As a result, an early-stopping condition valid for one scoring rule may not be valid for another. Implementations can use conservative bounds, heuristic stopping, or library-specific criteria. These are search semantics, not properties guaranteed by the underlying autoregressive model.
Minimum-length constraints add another interaction. If the end-of-sequence token is blocked before a threshold, short candidates cannot terminate even if the model assigns that token high probability. Maximum-length limits impose the opposite boundary by forcing search to stop or finalize according to implementation rules. A length penalty operates inside these externally imposed limits rather than replacing them.
Score comparability requires the full decoding contract
A reported beam score is meaningful only with its scoring convention. Raw cumulative log-probability, mean log-probability, and parameterized length-adjusted scores are different quantities. Comparing numeric scores across systems without the corresponding formula can produce a false equivalence even when both systems label the value as a sequence score.
The same applies to output differences. If two beam-search implementations disagree, the model is only one possible source. Beam width, score transformation, the stage at which that transformation is applied, end-of-sequence treatment, pruning order, and stopping criteria can all change the selected sequence without changing model parameters.
Length normalization is therefore best treated as part of the decoder’s objective, not as a cosmetic correction after generation. Once it participates in pruning or completion ranking, it helps define which sequence the search procedure is attempting to return.