Sampling

Control LLM Randomness with Temperature, Top-K, and Top-P

A language model does not normally produce a single inevitable next token. Given a prefix, it assigns scores to many possible tokens. A decoding algorithm then decides how to turn those scores into the next output. That last step matters. If you sample too freely, a model can drift into unlikely continuations. If you restrict sampling too aggressively, outputs can become repetitive or lose useful variation. Parameters such as temperature, top-k, and top-p control different parts of this trade-off, so treating them as interchangeable “creativity settings” leads to confusing results.