Use Predictive Entropy to Detect Uncertain Classifications

A classifier can return the same predicted label for two inputs while being much less certain about one of them. If an application only keeps the winning label, that difference disappears.

Predictive entropy gives you a compact way to preserve it. It summarizes how spread out a classifier’s predicted probability distribution is: concentrated probability produces low entropy, while probability spread across several classes produces higher entropy.

That makes entropy useful for tasks such as routing ambiguous cases to human review, inspecting difficult examples, or comparing uncertainty patterns across slices of data. It isn’t a guarantee that a prediction is correct, though. This article builds the calculation from a small example and shows how to use the signal without treating it as more trustworthy than the underlying probabilities.

The mental model: uncertainty is about the whole distribution

Suppose a three-class support-ticket classifier predicts billing, technical, or account.

For ticket A it returns:

billing     0.96
technical   0.03
account     0.01

For ticket B it returns:

billing     0.40
technical   0.35
account     0.25

Both tickets would be labeled billing by an argmax decision. They are not equally ambiguous. Ticket A has nearly all probability mass on one class. Ticket B has plausible alternatives competing with the winner.

You could look only at the maximum probability, and for many systems that is a useful baseline. Predictive entropy answers a slightly different question: how dispersed is the entire probability distribution? It accounts for every class rather than only the largest value.

For a categorical distribution with class probabilities p_1 ... p_K, Shannon entropy is:

H(p) = -sum_i p_i * ln(p_i)

Using the natural logarithm measures entropy in nats. Using a base-2 logarithm measures it in bits. Either base is fine if you use it consistently when setting thresholds and comparing values.

Calculate predictive entropy with a small example

Take ticket B:

p = [0.40, 0.35, 0.25]

Its entropy is:

H(p)
= -(0.40 * ln(0.40)
  + 0.35 * ln(0.35)
  + 0.25 * ln(0.25))
≈ 1.081 nats

Ticket A has much lower entropy:

p = [0.96, 0.03, 0.01]
H(p) ≈ 0.190 nats

For three classes, the largest possible entropy occurs at the uniform distribution:

p = [1/3, 1/3, 1/3]
H_max = ln(3) ≈ 1.099 nats

So ticket B is close to the maximum uncertainty available to a three-class distribution, while ticket A is strongly concentrated.

A minimal implementation is straightforward:

import math


def entropy(probabilities):
    return -sum(p * math.log(p) for p in probabilities if p > 0)

print(entropy([0.96, 0.03, 0.01]))  # about 0.190
print(entropy([0.40, 0.35, 0.25]))  # about 1.081

The p > 0 condition handles the mathematical convention that the contribution of a zero-probability outcome is zero. It also avoids evaluating log(0).

In production, probability vectors should still be validated. Values should be finite and non-negative, and they should sum to approximately one. If your model exposes logits rather than probabilities, apply the appropriate softmax before calculating categorical entropy.

Normalize entropy when class counts differ

Raw entropy depends on the number of possible classes. A uniform distribution over 3 classes has maximum entropy ln(3), while a uniform distribution over 100 classes has maximum entropy ln(100). Comparing those raw values directly can be misleading.

For a fixed set of K classes, you can normalize entropy to the interval from 0 to 1:

H_normalized(p) = H(p) / ln(K)

With three classes, ticket B becomes:

1.081 / ln(3) ≈ 0.984

A value near 0 means the distribution is concentrated. A value near 1 means it is close to uniform.

Normalization is useful when dashboards or evaluation pipelines compare tasks with different class counts. It does not make two classification problems automatically comparable. A 20-class taxonomy with many near-duplicate labels may have very different ambiguity from a clean 3-class problem, even if both produce the same normalized entropy.

Turn entropy into an application decision

Entropy becomes useful when it changes what your system does. Consider a support workflow where low-risk, clear tickets can be routed automatically and ambiguous tickets should be reviewed.

A simple policy might look like this:

model probabilities
       |
       v
calculate entropy
       |
       +---- low entropy ----> automatic route
       |
       +---- high entropy ---> human review

The difficult part is not calculating entropy. It is choosing what counts as “high.”

Do not choose a threshold because a normalized value such as 0.7 looks intuitively uncertain. Choose it against validation data that represents the deployment setting. For each candidate threshold, measure the outcomes the application actually cares about, such as:

• how many cases are sent to review;
• the error rate among cases accepted automatically;
• the kinds of errors that still pass the threshold;
• review volume for important data slices.

This turns the threshold into an operational trade-off. Lowering it may catch more ambiguous cases but increase review cost. Raising it may reduce review volume while allowing more model mistakes through.

The right operating point depends on the cost of those outcomes. A low-stakes content tagger and a system that routes expensive manual work should not inherit the same threshold merely because they use the same entropy formula.

Entropy and maximum probability are related but not identical

A common alternative is to use 1 - max(p) as an uncertainty score. It is simple, cheap, and often worth testing first.

The two measures can disagree because maximum probability ignores how the remaining probability mass is distributed. Compare these four-class predictions:

A = [0.60, 0.40, 0.00, 0.00]
B = [0.60, 0.14, 0.13, 0.13]

Both have maximum probability 0.60, so a max-probability rule treats them as equally uncertain. Their entropy differs:

H(A) ≈ 0.673 nats
H(B) ≈ 1.112 nats

Prediction B spreads its remaining mass across three alternatives, so entropy reports greater dispersion.

That extra information is useful when ambiguity among several alternatives matters. It is not automatically better for detecting errors. The practical question is which score separates acceptable and unacceptable predictions on your validation data. If maximum probability performs just as well for your decision, its simpler interpretation may be preferable.

High entropy and wrong predictions are not the same thing

The most important limitation is easy to miss: predictive entropy measures uncertainty expressed by the model’s output distribution. It does not independently know whether the model is correct.

A model can be confidently wrong:

true class: technical
prediction:
billing     0.98
technical   0.01
account     0.01

This prediction has low entropy even though it is wrong. An entropy threshold would probably accept it.

That can happen when the model is miscalibrated, encounters distribution shift, learns a spurious feature, or sees an input unlike the data on which its probability behavior was validated. Entropy cannot repair those problems by itself.

The reverse also occurs. A high-entropy prediction can still be correct. Some inputs are genuinely ambiguous, and a model that distributes probability across plausible labels may be behaving reasonably.

For this reason, evaluate entropy as an error-detection or routing signal, not as a substitute for accuracy, calibration, or task-specific evaluation.

Calibration changes how useful the threshold is

Suppose two classifiers have similar accuracy, but one routinely produces extreme probabilities and the other is more conservative. Their entropy distributions can look very different. A threshold learned for one model may therefore make little sense for the other.

Probability calibration asks whether predicted probabilities correspond sensibly to observed outcomes. Calibration and entropy are different concepts: entropy describes the shape of one predictive distribution, while calibration is a property evaluated over many predictions.

If you use entropy to drive decisions, inspect calibration as part of model evaluation. When a model is replaced, fine-tuned, quantized, or otherwise changed, revalidate the entropy threshold rather than assuming the old operating point still holds.

The same rule applies when the input distribution changes. A threshold chosen on clean benchmark data may produce an unexpected review rate on short messages, a new language, a new customer segment, or a changed label taxonomy.

Watch for large and structured label spaces

Entropy is easiest to interpret when the classes are mutually exclusive choices in one categorical distribution. Real systems can be messier.

In multi-label classification, each label may have an independent sigmoid probability rather than participating in one softmax distribution. Applying the categorical entropy formula to those values as though they summed to one is incorrect. You can calculate binary entropy for each label, but how those per-label uncertainties should be combined depends on the application.

Hierarchical taxonomies also need care. A model may split probability among several sibling labels that all lead to the same downstream action. Raw entropy sees that spread; the application may not care about it. Sometimes uncertainty over the final action is more useful than uncertainty over every leaf class.

Large label spaces create another practical issue: normalized entropy can be influenced by a long tail of tiny probabilities. Before adopting it, compare the score with simpler alternatives and inspect real examples around the proposed threshold.

Common mistakes when using predictive entropy

Treating entropy as a correctness probability. An entropy of 0.2 does not mean a prediction has an 80% chance of being correct. Entropy and correctness probability are different quantities.

Setting thresholds without validation data. The numerical scale is defined by the formula, but a useful operational cutoff comes from your model, label space, data, and error costs.

Comparing raw entropy across different class counts. The maximum grows as ln(K). Normalize when that comparison is meaningful, then still account for differences between tasks.

Calculating categorical entropy from arbitrary scores. Logits, independent sigmoid outputs, and unnormalized scores are not categorical probability distributions. Convert or handle them according to the model’s output semantics.

Assuming low entropy means the input is familiar. A model can produce a sharp distribution on out-of-distribution data. If detecting unfamiliar inputs matters, evaluate methods designed for that goal rather than relying on entropy alone.

When predictive entropy is a good fit

Predictive entropy is a useful starting point when your classifier already produces a categorical probability distribution and the application benefits from distinguishing concentrated predictions from ambiguous ones. It requires no additional model and is easy to compute after inference.

Use it as one feature in a decision process, then validate that process against real errors and operational costs. For some applications, maximum probability is enough. For others, a learned rejector, an ensemble, conformal methods, or explicit out-of-distribution detection may better match the requirement. Those approaches answer different questions and usually add their own data or compute requirements.

Make uncertainty actionable

Keeping only the predicted class throws away information the model already produced. Predictive entropy turns the full class distribution into one interpretable dispersion score, which can help expose ambiguous classifications and support review policies.

The practical next step is small: calculate entropy on an existing validation set, sort predictions from low to high entropy, and inspect both the error rate and the examples across that range. If errors become meaningfully more common as entropy rises, you have evidence for using it as a routing signal. If they do not, choose a different uncertainty signal instead of forcing entropy into the system.