Probability Calibration

Evaluate Probabilistic Classifiers with the Brier Score

Two classifiers can produce the same predicted labels and the same accuracy while assigning very different probabilities to those labels. A system that emits 0.51 for every correct binary decision is not making the same probabilistic claim as one that emits 0.99, even though thresholded accuracy may treat them identically. The Brier score keeps that distinction visible. It measures squared error between predicted probabilities and observed outcomes, so both the selected class and the probability assigned to each outcome affect the result. This makes it useful when downstream code consumes probabilities for ranking, thresholds, abstention, or expected-cost decisions.