https://nalar.dev/tags/probability/ Recent content in Probability on Nalar Hugo en-us Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/compare-model-distributions-with-kl-divergence/ Sun, 06 Sep 2026 00:00:00 +0700 https://nalar.dev/compare-model-distributions-with-kl-divergence/ <p>AI systems often produce probability distributions rather than single answers. A classifier assigns probabilities to classes, a language model assigns probabilities to possible next tokens, and a teacher model can provide a soft target distribution for a smaller student. In all of these cases, developers need a way to ask: <strong>how different is one probability distribution from another?</strong></p> <p><strong>Kullback-Leibler divergence</strong>, usually shortened to <strong>KL divergence</strong>, is one answer. It measures how much a comparison distribution <code>Q</code> differs from a reference distribution <code>P</code>, with the differences weighted by what <code>P</code> considers important.</p>