A language model assigns a probability to each next token, but applications often need to compare complete candidate sequences. A reranker may choose among generated answers. A decoder may keep several partial hypotheses. An evaluator may compare alternative completions under the same prompt.

The obvious approach is to multiply each candidate’s token probabilities, or equivalently add their log probabilities. That gives the probability the model assigns to the whole continuation. It also creates an important bias: every additional token contributes a probability no greater than 1, so longer sequences usually accumulate lower raw scores even when their individual tokens are highly plausible.

Length normalization changes the comparison from total log probability toward average log probability per generated token. It is useful when candidates of different lengths should compete without raw sequence length dominating the score. It is not a universal quality metric, however, and careless use can create the opposite problem by rewarding unnecessarily long continuations.

This article builds the idea from a small numerical example, explains what the normalized score means, and shows how to decide whether it belongs in a generation pipeline.

Start with the probability of a complete continuation

Suppose a model receives the prompt:

Complete the status message: Deployment

Consider two candidate continuations:

A: succeeded
B: succeeded without errors

An autoregressive language model scores a continuation one token at a time. If y_1, ..., y_T are the generated tokens and x is the prompt, the sequence probability is

P(y | x) = product over t of P(y_t | x, y_1, ..., y_(t-1))

Products of many small probabilities are inconvenient numerically, so implementations normally work with log probabilities. The logarithm turns the product into a sum:

log P(y | x) = sum over t of log P(y_t | x, y_<t)

Assume, for a simplified teaching example, that candidate A is one token with probability 0.60:

A: log score = ln(0.60) = -0.511

Candidate B contains three tokens whose conditional probabilities are 0.80, 0.80, and 0.80:

B: log score = ln(0.80) + ln(0.80) + ln(0.80)
             = 3 * -0.223
             = -0.669

The raw sequence score prefers A because -0.511 is greater than -0.669. Yet every token in B had a higher conditional probability than the single token in A.

Nothing is mathematically wrong. The two scores answer a specific question: which exact complete token sequence receives more probability mass from the model? A longer exact sequence has more conditional factors to multiply, so it often receives less probability mass.

The problem appears only when the application wants the score to answer a different question, such as: which candidate is more plausible per generated token?

Normalize by the number of generated tokens

The simplest length-normalized score divides total log probability by the number of scored tokens:

average_log_prob(y) = log P(y | x) / T

For the previous example:

A: -0.511 / 1 = -0.511
B: -0.669 / 3 = -0.223

Now B ranks higher. The normalization removes the direct accumulation of three negative log-probability terms and compares the candidates on their average token-level log probability.

There is another useful interpretation. Exponentiating the average log probability gives the geometric mean of the token probabilities:

exp(average_log_prob) = (product of token probabilities)^(1/T)

For the example, that produces 0.60 for A and 0.80 for B. This does not mean B has an 80% probability of being the correct answer. It means its scored tokens have a geometric-mean conditional probability of 0.80 under this model and tokenization.

That distinction matters. A language-model score describes the model’s probability distribution over token sequences. It is not automatically a calibrated probability of factual correctness, usefulness, safety, or task success.

Decide exactly which tokens belong in the score

Length normalization is only meaningful when the numerator and denominator refer to the same scored region.

For prompt-conditioned generation, the usual comparison is over generated continuation tokens, not prompt tokens:

prompt:      fixed context shared by candidates
continuation: tokens whose conditional log probabilities are summed
T:            number of scored continuation tokens

Including the same prompt contribution in every candidate can distort normalized scores because the shared constant is then divided by different candidate lengths. In many causal language-model APIs, prompt tokens and generated tokens are exposed separately; in lower-level implementations, the developer must align shifted logits and labels correctly.

Special tokens also require an explicit policy. If an end-of-sequence token is part of the model’s probability for terminating a candidate, including it can be reasonable. If one candidate is scored with an end token and another is truncated before termination, the scores are not directly describing the same event.

The practical rule is not that one convention is universally correct. It is that all candidates in a comparison must use a consistent scoring boundary, and the boundary should match the decision the application is making.

Understand why raw scores prefer shorter hypotheses

Log probabilities from a softmax are at most zero because probabilities are at most 1. Appending another ordinary token therefore adds another non-positive term:

old score = -2.1
next token log probability = -0.4
new score = -2.5

The new complete sequence cannot have a higher raw log probability than its prefix. This property is expected for joint probabilities: the event represented by a specific longer continuation is more restrictive than its prefix.

During search, however, that property can interact with termination. If a decoder compares completed hypotheses with raw cumulative log probability, short completed sequences may be attractive simply because they have accumulated fewer negative terms.

Length normalization reduces that pressure, but it changes the objective. The decoder is no longer ranking candidates purely by their joint sequence probability. It is ranking them by a transformed score chosen to better match the application’s preference over lengths.

That is why length normalization should be treated as a decision rule, not as a correction that makes model probabilities more mathematically valid.

Use a tunable length penalty when averaging is too strong

Dividing by T is full average-log-probability normalization. Some generation systems instead use a tunable exponent:

score(y) = log P(y | x) / T^alpha

The parameter alpha controls how strongly length affects ranking:

  • alpha = 0 gives the unnormalized cumulative log probability.
  • alpha = 1 gives the average log probability per token.
  • values between 0 and 1 interpolate between those objectives.

This formula is useful as a mental model, but production libraries do not all define a parameter named length_penalty with this exact formula. Some use different denominators or transformations. A value copied from one decoder therefore may not mean the same thing in another implementation.

Before tuning such a parameter, inspect the actual scoring equation in the library or service being used. The semantic behavior of the formula matters more than the parameter name.

Compare candidates only when the comparison is well defined

Length-normalized model scores are most useful when candidates share important conditions.

Keep the prompt fixed

If two continuations are scored under different prompts, their normalized log probabilities answer different conditional questions. Comparing them may still be useful for a carefully designed experiment, but the score difference cannot be attributed only to continuation quality.

For candidate selection under one request, score every candidate under the same prompt and model state.

Keep tokenization fixed

The denominator counts tokens, not words or characters. Different tokenizers can split the same text differently, so average log probability is tokenizer-dependent.

Even under one tokenizer, two semantically similar strings can have different token counts. That does not make the metric invalid, but it means the unit is specifically a model token. Avoid presenting it as a tokenizer-independent measure of linguistic quality.

Keep model scoring conditions fixed

Scores can change with model weights, system instructions, chat templates, preceding context, and other inputs that affect next-token probabilities. Compare candidates using the same scoring setup unless changing that setup is itself the experiment.

Do not confuse normalized likelihood with answer quality

A common mistake is to use average log probability as if it were a general-purpose reward model.

Imagine a support assistant choosing between:

A: Restart the service.
B: Restart the service after confirming no migration is running.

The model could assign A a higher normalized likelihood because it is a common, fluent continuation. That does not establish that A is safer or more appropriate for the current system state.

Language-model likelihood is influenced by how probable a token sequence is under the training and conditioning distribution. Task quality can depend on properties that likelihood does not directly encode: factual grounding, instruction compliance, completeness, harmlessness, business rules, or user preference.

For those properties, use task-specific validation or scoring where possible. For example, structured checks can verify required fields, retrieval-based checks can test grounding, and a separately evaluated preference or reward model can rank qualities that base-model likelihood does not represent reliably.

Length-normalized likelihood can be one signal in such a system. It should not silently become the definition of quality.

Watch for the opposite bias: rewarding verbosity

Raw cumulative probability tends to penalize length. Full average normalization can remove so much of that penalty that longer candidates become competitive by continuing with easy, predictable tokens.

Suppose two answers communicate the same fact, but one adds a generic closing sentence that the model predicts with very high probability. Under average log probability, those easy extra tokens can improve the overall average.

This creates a practical failure mode: optimizing the normalized score alone may prefer fluent padding rather than concise completion.

Several controls can help, depending on the task:

  • constrain acceptable output length when the format has a natural bound;
  • tune the normalization strength on representative validation examples;
  • score semantic or task-specific requirements separately;
  • compare only candidates produced under similar stopping rules;
  • evaluate the final selection metric against human or application outcomes rather than assuming likelihood ranking is sufficient.

A good length penalty is therefore task-dependent. The desired setting for machine translation, short classification labels, free-form answers, and code generation need not be the same.

Treat stopping as part of sequence scoring

Generation length is determined not only by a length penalty but also by how candidates terminate.

A candidate that naturally emits an end-of-sequence token represents a different model event from a candidate forcibly cut off at a maximum token limit. If both are assigned scores, record whether each candidate actually terminated. Otherwise a truncated fragment may look competitive even though the model never assigned probability to ending there.

Minimum-length rules can also interact with scoring by preventing early termination, while maximum-length rules can remove longer hypotheses before their relative score matters. Sampling parameters affect which candidates are generated in the first place, even if a later reranking stage scores them with model log probabilities.

For this reason, debug sequence selection as a pipeline:

candidate generation
    -> stopping rules
    -> token log probabilities
    -> sequence aggregation
    -> length adjustment
    -> task-specific checks or reranking
    -> final selection

Looking only at the final normalized number can hide a problem that actually originates in candidate generation or stopping.

Evaluate the scoring rule on the decision you care about

The most reliable way to choose a sequence-scoring rule is to evaluate the resulting decisions.

Build a validation set containing realistic prompts and multiple plausible candidates. For each candidate, store at least:

raw cumulative log probability
number of scored tokens
normalized score
termination status
task-specific quality label or metric

Then compare selection rules. Does raw log probability choose answers that are too short? Does full average normalization choose verbose answers? Does an intermediate length penalty improve the application metric without producing undesirable length shifts?

Also inspect results by length bucket. An aggregate win rate can hide a rule that works well for medium outputs but fails badly for very short or very long ones.

If the application already constrains every candidate to the same token length, normalization will not change their ranking because every score is divided by the same factor. In that case, adding a length penalty only complicates the system.

Know when a simpler method is better

Length-normalized sequence scoring is useful when you have multiple variable-length candidates from the same model and need likelihood to contribute to their ranking. It is especially relevant to search procedures and candidate reranking where raw cumulative probability creates an unwanted preference for brevity.

It is less useful when the task has a direct deterministic validator. If generated SQL either passes a schema-aware check or fails it, that signal may be more valuable than a small likelihood difference. If an answer must cite retrieved evidence, grounding checks should carry more weight than fluency. If all outputs have fixed length, raw and normalized rankings are equivalent.

The simplest scoring rule that matches the real decision is usually easier to evaluate and maintain.

Conclusion

A language model’s raw sequence log probability is the sum of its token log probabilities. Because each additional token contributes another non-positive term, raw scores naturally favor shorter exact sequences when candidates of different lengths compete.

Dividing by token count converts that total into average log probability, reducing the direct effect of length. Tunable penalties can interpolate between cumulative and fully normalized scoring, but their exact definitions vary across implementations.

The key engineering lesson is to choose the score for the decision you actually need. Define consistent scoring boundaries, account for tokenization and stopping, and validate the ranking against task outcomes. Length normalization is a useful decoding tool when length bias is the problem; it is not a substitute for measuring whether the generated answer is actually good.

Artificial Intelligence Language Models Model Inference Decoding