Evaluate Probabilistic Classifiers with the Brier Score

Two classifiers can produce the same predicted labels and the same accuracy while assigning very different probabilities to those labels. A system that emits 0.51 for every correct binary decision is not making the same probabilistic claim as one that emits 0.99, even though thresholded accuracy may treat them identically.

The Brier score keeps that distinction visible. It measures squared error between predicted probabilities and observed outcomes, so both the selected class and the probability assigned to each outcome affect the result. This makes it useful when downstream code consumes probabilities for ranking, thresholds, abstention, or expected-cost decisions.

The score operates on probabilities, not just labels

For a binary event with predicted probability p and observed outcome y in {0, 1}, the Brier score for one prediction is

BS = (p - y)^2

A dataset score is the mean of those squared errors. Lower values indicate smaller probability error under this definition.

Consider a positive outcome where y = 1. Predictions of 0.9 and 0.6 both select the positive class under a 0.5 threshold, but their Brier contributions differ:

(0.9 - 1)^2 = 0.01
(0.6 - 1)^2 = 0.16

Accuracy records both as correct. The Brier score records that the first probability is closer to the observed outcome.

The same squared penalty also makes confident errors expensive. If the event does not occur, so y = 0, a prediction of 0.9 contributes 0.81, while 0.6 contributes 0.36. A model cannot improve this score merely by becoming more confident; increased confidence helps only when the probability moves toward the realized outcome.

Multiclass definitions need an explicit convention

For mutually exclusive classes, a common multiclass form compares the complete probability vector p with a one-hot outcome vector o:

BS = sum_k (p_k - o_k)^2

Some definitions divide this sum by the number of classes or use another normalization. Those conventions change the numerical range without changing the core squared-error structure. Reported Brier scores are therefore meaningful only when the scoring convention is clear.

The full-vector form also differs from evaluating only the probability assigned to the predicted class. Suppose the true class is A and two three-class models emit:

model X: [0.60, 0.39, 0.01]
model Y: [0.60, 0.20, 0.20]

Both assign 0.60 to the true class, yet the remaining probability mass is distributed differently. Under the unnormalized multiclass form, their scores are

X: (0.60 - 1)^2 + 0.39^2 + 0.01^2 = 0.3122
Y: (0.60 - 1)^2 + 0.20^2 + 0.20^2 = 0.2400

The metric evaluates the complete categorical probability claim rather than only the winning probability.

A proper score discourages strategic probability distortion

The Brier score is a proper scoring rule: when outcomes follow a probability distribution, the expected score is minimized by reporting that distribution rather than a deliberately distorted one. This property matters when a metric is used to select probabilistic models.

A metric based only on classification accuracy has no comparable pressure on probability values once the class ordering is correct. Moving a binary prediction from 0.55 to 0.95 changes nothing if both remain on the correct side of the decision threshold. The Brier score continues to respond because those values represent different probability estimates.

Properness does not mean a finite evaluation set can reveal the true data-generating probabilities. Sampling variation, distribution shift, label noise, and model misspecification still affect observed scores. The property describes the incentive created by the scoring rule in expectation, not a guarantee about any particular test sample.

Calibration and resolution both affect the result

A low Brier score is not synonymous with calibration. Calibration concerns agreement between stated probabilities and outcome frequencies. The Brier score also rewards predictions that separate cases with different event rates.

A constant predictor illustrates the distinction. If an event occurs in roughly one quarter of an evaluation population, a model that emits 0.25 for every case can be well aligned with that aggregate event rate. It still provides no case-level separation between higher-risk and lower-risk examples. A model that assigns distinct probabilities to informative groups can achieve a lower Brier score when those distinctions correspond to outcome differences.

This means a Brier score should not be described as a pure calibration metric. It combines aspects of probability reliability and discriminatory resolution into a single loss. When the source of a score difference matters, calibration plots or a suitable score decomposition can provide information that the scalar value does not contain by itself.

Class prevalence changes the baseline

Brier scores should be compared on the same target definition and evaluation distribution. The event prevalence influences the score attainable by simple reference predictors.

For a binary event with prevalence r, a constant predictor that emits r for every case has expected Brier score

r * (1 - r)^2 + (1 - r) * r^2

which simplifies to

r * (1 - r)

That baseline changes as prevalence changes. A score of 0.08 on one population and 0.08 on another does not establish equal model quality if the populations have substantially different event rates or represent different prediction tasks.

This also makes careless comparisons across filtered datasets misleading. Removing easy negatives, changing the sampling ratio, or evaluating a different time period can alter the reference difficulty even when model parameters stay fixed.

Probability quality and decision quality are separate questions

The Brier score evaluates probability estimates under squared error. A deployed decision often uses a threshold or a cost function that values some errors more than others. Those objectives are related but not interchangeable.

For example, a classifier can improve its Brier score while leaving all probabilities on the same side of an operational threshold, producing no change in binary actions. Conversely, a small probability adjustment near that threshold can change many actions while barely moving the average Brier score.

Model evaluation is clearer when the metric matches the artifact being assessed. Use the Brier score to examine probabilistic predictions, then evaluate thresholded decisions with metrics or expected costs that represent the actual decision rule. Treating one scalar as a substitute for both questions hides information that the application may depend on.