Inference

Control LLM Randomness with Temperature and Top-p

Large language models usually generate text one token at a time. At each step, the model assigns scores to possible next tokens, those scores become probabilities, and a decoding strategy chooses what comes next. Two common controls in that process are temperature and top-p. They are often described as creativity settings, but that description is incomplete. They change how the model samples from its probability distribution, which affects repeatability, diversity, and the chance of selecting lower-probability tokens.