https://nalar.dev/tags/classification/ Recent content in Classification on Nalar Hugo en-us Tue, 01 Sep 2026 00:00:00 +0700 https://nalar.dev/calibrate-classification-probabilities-decision-thresholds/ Tue, 01 Sep 2026 00:00:00 +0700 https://nalar.dev/calibrate-classification-probabilities-decision-thresholds/ <p>A classifier can rank examples well while producing poor probability estimates. If predictions drive cost-sensitive decisions, triage, or risk thresholds, the difference matters. A score of <code>0.8</code> is useful as a probability only when similarly scored examples are positive about 80% of the time under the deployment distribution.</p> <h2 id="separate-discrimination-from-calibration">Separate discrimination from calibration</h2> <p>Metrics such as ROC AUC primarily measure ranking. Calibration asks whether predicted probabilities agree with observed frequencies. A model can have strong AUC and still be overconfident or underconfident.</p> https://nalar.dev/probability-calibration-for-classification-models/ Tue, 01 Sep 2026 00:00:00 +0700 https://nalar.dev/probability-calibration-for-classification-models/ <p>A classifier can rank examples correctly while producing probabilities that are poor estimates of real-world likelihood.</p> <p>If a model assigns 0.8 probability to many comparable cases, calibration asks whether roughly 80% of those cases are actually positive. This matters whenever probabilities drive decisions such as pricing, triage, alert thresholds, expected value, or human review.</p> <h2 id="discrimination-and-calibration-are-different">Discrimination and calibration are different</h2> <p>Metrics such as ROC AUC evaluate how well a model ranks positive examples above negative ones. They do not require predicted probabilities to match observed frequencies.</p>