A token-level bias is usually applied to model scores before probabilities are normalized. That placement matters. Adding a constant to one token’s logit changes its odds relative to every token that does not receive the same constant, even though the model parameters and hidden state remain unchanged.
The mechanism is simple, but its operational effect depends on the rest of the decoding pipeline.
Additive bias acts on score differences
For a vocabulary with logits (z_1, \ldots, z_V), softmax assigns token (i) the probability
[ p_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}. ]
Suppose a decoder adds bias (b_i) before softmax. The adjusted distribution is
[ p_i’ = \frac{\exp(z_i + b_i)}{\sum_j \exp(z_j + b_j)}. ]
For two tokens (i) and (k), their probability ratio becomes
[ \frac{p_i’}{p_k’} = \exp\big((z_i-z_k) + (b_i-b_k)\big). ]
Only relative bias matters to this ratio. Adding the same constant to every vocabulary logit leaves the softmax distribution unchanged. A bias applied to one token, in contrast, directly changes its log-odds against unmodified alternatives.
This also means a numeric bias has no context-independent probability interpretation. The resulting probability depends on the original logits and on all other candidates in the normalization term.
Finite bias is not a hard constraint
A positive finite bias raises a token’s score but does not mathematically guarantee selection. Another token can still have a larger adjusted score, and stochastic sampling can still choose another candidate when multiple tokens retain nonzero probability.
Likewise, a negative finite bias suppresses a token without making its probability exactly zero under ideal real-valued softmax. Systems that need an exclusion rule generally require a hard mask or an implementation-specific mechanism that removes the candidate from consideration.
Very large magnitudes introduce another boundary: production runtimes may clamp values, use reduced precision, represent masks with special sentinel values, or reorder transformations for numerical stability. Those details are properties of the serving implementation rather than of the softmax equation itself.
Tokenization determines the unit being biased
A logit index refers to a vocabulary token, not directly to a word, phrase, or semantic concept. Text that appears as one word to a user can map to several token IDs depending on the tokenizer, surrounding whitespace, capitalization, or byte-level representation.
Biasing one token ID therefore affects that token wherever it is a valid next-token candidate. It does not automatically target every textual spelling or every token sequence that can render the same visible string.
Multi-token phrases expose the boundary more sharply. A bias on the first token can alter the probability of entering a phrase, but subsequent phrase tokens are scored on later decoding steps from a changed context. Token-level bias alone does not express an arbitrary sequence constraint.
Pipeline order changes the effective candidate set
Decoders often combine several score transformations and filters. Temperature scaling, repetition penalties, vocabulary masks, top-k selection, nucleus filtering, and other runtime-specific processors may all participate.
These operations are not generally interchangeable. For example, adding a bias before a top-k filter can move a token into or out of the retained set. Adding the same bias after a token has already been removed by a hard filter cannot restore it unless the implementation explicitly reconstructs that candidate set.
Temperature also changes the effective strength of a pre-temperature bias. If adjusted logits are divided by temperature (T),
[ \tilde{z}_i = \frac{z_i+b_i}{T}, ]
then the pairwise log-odds contribution from the bias becomes ((b_i-b_k)/T). An API that defines bias at another stage can have different semantics, so the documented processor order belongs to the interface contract.
Bias can alter later model state indirectly
The bias itself does not modify model weights or the current hidden state. Once it changes which token is selected, however, that selected token becomes part of the next decoding context. Future logits can then differ because autoregressive generation conditions on prior output.
This separates two effects that are easy to conflate. At the current step, bias is an external score transformation. At later steps, any changed token can redirect the model trajectory through ordinary autoregressive conditioning.
The distinction matters when diagnosing output differences. A large divergence several tokens later does not imply that the bias was directly applied to those later logits.
Interface semantics matter more than the parameter name
A field named logit_bias suggests additive score adjustment, but a concrete API still defines which token identifiers it accepts, the allowed numeric range, the stage where adjustment occurs, and its interaction with masks or sampling controls. Those details should be treated as implementation-specific unless the interface documents them.
At the mechanism level, the stable boundary is narrower: additive bias modifies relative scores before whatever subsequent operations the decoder performs. It is suitable for shifting token preference when its placement is known. It is not, by itself, a general grammar, phrase constraint, or guarantee that a visible string must appear or disappear.