https://nalar.dev/tags/top-p-sampling/ Recent content in Top-P Sampling on Nalar Hugo en-us Tue, 22 Sep 2026 00:00:00 +0000 https://nalar.dev/top-p-sampling-rebuilds-its-candidate-set-at-every-token/ Tue, 22 Sep 2026 00:00:00 +0000 https://nalar.dev/top-p-sampling-rebuilds-its-candidate-set-at-every-token/ <p>Top-p sampling does not keep a fixed shortlist of tokens throughout generation. At each decoding step, the model produces a new logit vector, that vector becomes a probability distribution, and the sampler forms a new candidate set whose cumulative probability mass reaches the configured threshold.</p> <p>The consequence is easy to miss in serving code: the same <code>top_p</code> value can admit two tokens at one step and dozens at another. The parameter controls probability mass, not candidate count.</p>