https://nalar.dev/tags/calibration/ Recent content in Calibration on Nalar Hugo en-us Tue, 01 Sep 2026 00:00:00 +0700 https://nalar.dev/calibrate-classification-probabilities-decision-thresholds/ Tue, 01 Sep 2026 00:00:00 +0700 https://nalar.dev/calibrate-classification-probabilities-decision-thresholds/ <p>A classifier can rank examples well while producing poor probability estimates. If predictions drive cost-sensitive decisions, triage, or risk thresholds, the difference matters. A score of <code>0.8</code> is useful as a probability only when similarly scored examples are positive about 80% of the time under the deployment distribution.</p> <h2 id="separate-discrimination-from-calibration">Separate discrimination from calibration</h2> <p>Metrics such as ROC AUC primarily measure ranking. Calibration asks whether predicted probabilities agree with observed frequencies. A model can have strong AUC and still be overconfident or underconfident.</p>