Top-p sampling does not keep a fixed shortlist of tokens throughout generation. At each decoding step, the model produces a new logit vector, that vector becomes a probability distribution, and the sampler forms a new candidate set whose cumulative probability mass reaches the configured threshold.
The consequence is easy to miss in serving code: the same top_p value can admit two tokens at one step and dozens at another. The parameter controls probability mass, not candidate count.
The cutoff follows the current distribution
For a vocabulary with token probabilities p_1, p_2, ..., p_V, sort the tokens from highest to lowest probability:
p_(1) >= p_(2) >= ... >= p_(V)For a threshold q, top-p sampling retains the smallest leading set whose cumulative mass reaches at least q:
sum(i=1..k) p_(i) >= qThe exact boundary behavior can vary across implementations, especially around the token that crosses the threshold and any minimum-token rule. The central property remains the same: membership is derived from the current distribution rather than from a fixed value of k.
Consider two distributions with top_p = 0.90:
step A: [0.72, 0.19, 0.04, 0.03, 0.02]
step B: [0.24, 0.22, 0.18, 0.15, 0.12, 0.09]At step A, the first two tokens already carry 0.91 probability mass. At step B, reaching 0.90 requires substantially more candidates. A single threshold therefore adapts the candidate-set width to distribution concentration.
Logits make the set dynamic from one step to the next
Autoregressive decoding changes the model input after every selected token. The next forward pass produces a different logit vector because the prefix has changed. Any decoding transform applied before top-p filtering can also change the distribution presented to the cutoff.
Temperature is one example. When an implementation applies temperature to logits before softmax and then performs top-p filtering, changing temperature can alter how concentrated the probabilities are. A flatter distribution can require more tokens to reach the same cumulative mass; a sharper distribution can require fewer.
Other logit processors can have similar effects if they run before the top-p stage. Repetition controls, token bans, grammar constraints, or application-specific biasing can move probability mass or remove candidates. The resulting nucleus belongs to the processed distribution at that decoding step, not to an immutable ranking established when generation began.
This makes operation order part of decoding semantics. Two systems exposing parameters with the same names can produce different candidate sets if they apply transforms in a different order or use different boundary rules. Parameter equality alone does not establish equivalent sampling behavior.
Top-p and top-k constrain different quantities
Top-k sampling keeps a fixed maximum number of highest-ranked candidates before sampling. Top-p keeps enough high-ranked candidates to cover a probability-mass threshold. The two controls can therefore react differently to the same logits.
With a highly concentrated distribution, top-p may retain only a small set even when the vocabulary is large. With a diffuse distribution, it may retain a much larger set. Top-k does not expand in response to that diffusion once its configured count is reached.
When both controls are active, the effective set depends on the runtime’s filtering sequence and semantics. A common design applies multiple filters to the same processed logits, leaving only tokens that survive the combined restrictions, but an application should treat the specific library or serving runtime as authoritative for exact ordering.
This distinction matters for configuration portability. Copying top_p = 0.9 and top_k = 50 between runtimes does not by itself prove that the same token set will be sampled. Tokenizer vocabulary, model logits, prior processors, numerical details, minimum-candidate rules, and filter ordering can all affect the boundary.
Renormalization changes probabilities inside the retained set
After tokens outside the top-p set are excluded, sampling operates over the retained candidates. Conceptually, their remaining probabilities are normalized so they sum to one.
Suppose the retained raw probabilities are:
[0.55, 0.25, 0.12]Their total mass is 0.92. Sampling from only these candidates corresponds to normalized weights:
[0.55 / 0.92, 0.25 / 0.92, 0.12 / 0.92]The filter therefore does more than prevent low-ranked tokens from being selected. It also changes the conditional probability of every retained candidate relative to sampling from the unfiltered distribution.
Exact implementation details can differ. Some libraries represent removed logits as negative infinity and rely on a later softmax or sampling operation; others expose processors and warpers as separate stages. The semantic boundary to preserve is that excluded candidates contribute no sampling mass after filtering.
A stable threshold does not imply stable output diversity
Because candidate-set width depends on the current distribution, top_p is not a direct diversity quota. A threshold of 0.9 does not mean ten percent of the vocabulary is excluded, nor does it imply a stable number of alternatives per token.
The same setting can behave conservatively when one or two tokens dominate the distribution and admit a broader set when probability mass is spread across many plausible continuations. This adaptive property is the mechanism, not a side effect.
It also means aggregate generation behavior cannot be inferred from top_p in isolation. Temperature, model state, prompt tokens, logit processors, constraints, random seed handling, and the sampler implementation all participate in the sequence of distributions from which tokens are drawn.
For deterministic replay, recording only the top-p value is therefore insufficient. Reproducing a sampled sequence generally also requires the same model and relevant state, tokenized prefix, decoding transforms and their order, random-number state, and implementation behavior. Hardware and numerical execution can add further sensitivity near ranking or cutoff boundaries.
Observability should capture the changing boundary
A serving system that records only configured sampling parameters can miss useful evidence about actual decoding behavior. Candidate count after filtering, cumulative mass at the boundary, and the rank of the selected token can describe the sampler’s state more directly than the static threshold alone.
Such telemetry needs careful interpretation. Candidate count is not a model-quality metric, and comparing counts across different vocabularies or processing pipelines can be misleading. It is most useful for tracing configuration changes, detecting unexpected filter ordering, or explaining a divergence between two supposedly equivalent inference paths.
The practical boundary is precise: top-p specifies a cumulative probability target over the distribution available at a particular decoding step. It does not reserve a fixed subset of vocabulary entries. Any system that caches, audits, reproduces, or compares sampling decisions has to account for the fact that the candidate set is reconstructed as the logits change.