A decoder can favor or suppress a token without changing model weights. Add a constant to that token’s logit before softmax, and its probability changes relative to the rest of the vocabulary. The operation is simple, but its effect depends on where the bias enters the decoding pipeline and on every transformation that follows it.
This makes additive logit bias useful as an inference control, but not as a general semantic constraint. It changes a score used by the decoder. It does not rewrite the model’s internal representation or guarantee that a concept disappears from generated text.
The bias acts on scores before normalization
For a vocabulary with logits z_1, ..., z_V, ordinary softmax assigns
p_i = exp(z_i) / sum_j exp(z_j)Suppose the decoder adds a bias vector b before normalization:
p'_i = exp(z_i + b_i) / sum_j exp(z_j + b_j)A positive b_i raises token i relative to tokens receiving a smaller bias. A negative value lowers it. The denominator is recomputed from all adjusted logits, so changing one token also changes the normalized probabilities of the others even when their logits remain untouched.
The useful invariant is relative odds. For tokens i and k,
p'_i / p'_k = exp((z_i - z_k) + (b_i - b_k))Compared with the unbiased distribution, their odds ratio is multiplied by
exp(b_i - b_k)under this exact placement of the additive bias. The effect on relative odds therefore depends on the difference between two bias values, not on either value in isolation.
Adding the same constant to every vocabulary logit changes nothing after softmax. The shared offset cancels during normalization. A bias vector matters only through differences among its entries.
Probability change depends on the original distribution
A fixed positive bias does not imply a fixed increase in probability. The same additive shift can have very different normalized effects depending on the competing logits.
Consider one target token with original probability p and a bias c, while every other token receives zero bias. Its adjusted probability can be written as
p' = exp(c) p / (1 - p + exp(c) p)This expression shows two boundaries. First, the bias multiplies the target token’s odds against the rest by exp(c). Second, the resulting probability still depends on p. A token starting with tiny probability can remain unlikely after a moderate positive bias, while the same bias applied to an already competitive token can make it dominant.
The corresponding negative case behaves similarly. A finite negative bias reduces odds but does not mathematically force probability to zero. Exact exclusion requires a mechanism that removes the token from the candidate set or assigns it an effective score of negative infinity under the implementation’s numeric conventions.
Temperature changes the effective strength of a bias
Pipeline order matters when temperature scaling is present. If adjusted logits are divided by temperature T > 0 after the bias is added, the distribution is
p'_i = softmax((z_i + b_i) / T)and the bias difference contributes (b_i - b_k) / T to the log-odds difference. Lower T magnifies that contribution; higher T reduces it.
A different implementation could scale model logits first and add an unscaled bias afterward:
p'_i = softmax(z_i / T + b_i)Those two expressions are not equivalent unless the bias is scaled consistently. An API parameter named as a logit bias does not, by its name alone, establish its position relative to temperature, repetition penalties, vocabulary masks, or other processors. That ordering is an implementation property.
This distinction also affects reproducibility. Two serving stacks can accept numerically identical bias values yet produce different distributions if their processor order differs.
Candidate filtering can override the adjusted ranking
Sampling often contains stages beyond softmax. Top-k filtering can retain only a fixed number of high-scoring candidates. Top-p filtering can build a candidate set from cumulative probability mass. Vocabulary masks can remove tokens before sampling. Other processors can modify scores based on generated context.
A positive bias can move a token upward before these stages, but it does not guarantee that the token survives every later filter. Conversely, a token receiving no bias can disappear because another processor excludes it.
The exact interaction depends on ordering. If top-k selection occurs after bias application, the bias can change membership in the top-k set. If a hard vocabulary mask runs afterward, the mask can still eliminate the token regardless of its adjusted score. Treating all decoding controls as if they were one softmax operation hides these boundaries.
For debugging, the relevant artifact is the score-processing pipeline: raw model logits, each processor in order, the final candidate set, and the sampling distribution. Inspecting only the final sampled token loses the intermediate state needed to attribute a change to one processor.
Token controls are not text controls
Logit bias operates on token IDs. Text constraints operate on strings or semantic content. The two levels do not coincide.
A visible word can have several tokenizations depending on tokenizer vocabulary, leading whitespace, capitalization, punctuation, or surrounding bytes. Suppressing one token ID therefore does not necessarily suppress every sequence that renders the same text. A concept can also be expressed with synonyms or paraphrases whose tokens were never biased.
Positive bias has a related boundary. Favoring one token does not guarantee a particular word sequence, because the next decoding step receives a new model distribution conditioned on the token already emitted. Multi-token phrases require reasoning about a sequence of conditional distributions, not one static vocabulary adjustment.
For strict lexical constraints, a decoder needs a mechanism designed around allowed or disallowed token sequences, often with state that tracks partial matches. For semantic restrictions, token-level score shifts are an even weaker proxy because semantic equivalence is not represented by one fixed token set.
Extreme values meet finite-precision behavior
The equations above assume real arithmetic. Serving implementations use finite numeric types and typically apply numerically stable softmax transformations. Very large positive or negative adjustments can therefore encounter clipping, saturation, explicit bounds, or processor-specific handling.
A large positive bias also does not create probability greater than one; normalization redistributes mass across the candidate set. As the target score becomes dominant, its probability approaches one under an otherwise unchanged finite candidate set. A sufficiently negative finite score approaches zero without becoming an exact hard exclusion in the mathematical softmax.
For an external inference API, accepted bias ranges and handling of extreme values are API contracts rather than properties of softmax itself. Those details need to come from the implementation’s documentation or observed interface contract, not from the additive-bias equation.
Bias is a local decoding intervention
Additive logit bias has a narrow interpretation: it changes selected token scores at a particular decoding step. Under direct pre-softmax addition, differences in bias translate exactly into multiplicative changes in relative odds before other processors intervene.
That precision is useful because it separates a mathematical property from broader product behavior. Temperature, filtering, hard masks, tokenizer structure, context-dependent logits, and processor order can all change the eventual sampled sequence. A bias value is therefore best treated as one local intervention in the decoding pipeline, with its scope defined by the stage where it is applied.