When labels are expensive, training on every available example may be impractical. An active learning system tries to spend its labeling budget selectively: train a model on the labels already available, score unlabeled examples, request labels for useful examples, then retrain.

A common first idea is to label the examples with the highest predictive entropy. That can help, but entropy mixes together two different reasons for uncertainty. The model may be uncertain because it does not yet know enough, or because the input itself is genuinely ambiguous. More labels are most valuable for the first case.

BALD, short for Bayesian Active Learning by Disagreement, provides a useful mental model for separating those cases. Instead of asking only whether the final prediction is uncertain, BALD asks whether plausible versions of the model disagree about the prediction. This article explains that distinction, derives the score from a small example, and shows how to use it without treating uncertainty estimates as guarantees.

Start with two kinds of uncertainty

Imagine a classifier that routes support tickets into billing or technical.

Consider two unlabeled tickets:

A: "My card was charged twice after checkout."
B: "The service stopped after my account changed."

Suppose the current classifier assigns both tickets an average probability near 50% for each class. Predictive entropy therefore says both are highly uncertain.

But the reason can differ.

For ticket A, different plausible model parameters might produce very different predictions:

model sample 1: billing 0.90, technical 0.10
model sample 2: billing 0.55, technical 0.45
model sample 3: billing 0.15, technical 0.85

The models disagree. Observing the correct label may help determine which model beliefs fit the task better.

For ticket B, every model sample might be similarly uncertain:

model sample 1: billing 0.52, technical 0.48
model sample 2: billing 0.49, technical 0.51
model sample 3: billing 0.50, technical 0.50

Here the models agree that the ticket is ambiguous. A label is still useful as training data, but it may reveal less about which model parameters are appropriate.

This distinction is often described as epistemic uncertainty versus aleatoric uncertainty. Epistemic uncertainty comes from limited knowledge about the model and can, in principle, shrink as relevant data arrives. Aleatoric uncertainty comes from ambiguity or noise in the observation or target-generating process and may remain even with more data.

BALD is designed to prefer examples with the first kind of uncertainty: disagreement about what the model should believe.

Predictive entropy alone cannot expose disagreement

For a classification distribution p(y | x), predictive entropy is:

H[y | x] = -sum_y p(y | x) log p(y | x)

Entropy is low when one class receives nearly all the probability and high when probability is spread across classes.

For binary classification, a prediction near (0.5, 0.5) has high entropy regardless of how that average was produced. That is the key limitation for active learning.

These two situations can have the same average prediction:

case 1: individual models are confident but disagree
case 2: individual models are all uncertain in the same way

Predictive entropy sees only the average. BALD also examines the predictions before they are averaged.

BALD measures disagreement through mutual information

Let w represent uncertain model parameters and D the labeled training data observed so far. For an unlabeled input x, BALD scores the mutual information between the unknown label y and the model parameters:

BALD(x) = I[y, w | x, D]

A useful equivalent form is:

BALD(x)
  = H[y | x, D]
    - E_{p(w | D)} H[y | x, w]

The first term is the entropy of the model’s overall predictive distribution. It asks:

How uncertain is the prediction after accounting for uncertainty about the model?

The second term is the expected entropy of predictions from individual parameter settings. It asks:

How uncertain is a typical plausible model by itself?

Subtracting the second quantity from the first isolates disagreement among model beliefs.

If individual models are confident but choose different classes, the average prediction can be uncertain while the average individual entropy stays low. BALD is high.

If every individual model produces roughly the same uncertain prediction, both entropy terms are high and similar. Their difference is small.

That subtraction is the central mental model.

Work through a small numerical example

Use two equally weighted model samples for a binary classifier.

For an input x_A, suppose the predictions are:

sample 1: (0.9, 0.1)
sample 2: (0.1, 0.9)

Their mean is:

mean: (0.5, 0.5)

Using natural logarithms, the entropy of the mean is approximately:

H(0.5, 0.5) = 0.693 nats

Each individual distribution has entropy about 0.325 nats, so:

BALD(x_A) ~= 0.693 - 0.325 = 0.368 nats

Now consider x_B:

sample 1: (0.5, 0.5)
sample 2: (0.5, 0.5)

The entropy of the mean is still 0.693 nats. But each individual prediction also has entropy 0.693 nats:

BALD(x_B) = 0.693 - 0.693 = 0

Predictive entropy ranks the two inputs equally. BALD distinguishes them because only x_A contains disagreement among the sampled model beliefs.

The exact numerical value depends on the logarithm base, but ranking is unchanged when the same base is used consistently.

Approximate BALD with repeated predictive samples

The definition refers to a posterior distribution over model parameters, which is usually not available exactly for modern neural networks. Practical systems therefore need an approximation that can produce multiple plausible predictive distributions for the same input.

Examples include a Bayesian model with posterior samples, an ensemble of independently trained models, or a model for which stochastic inference is intentionally used as an uncertainty approximation. The quality of BALD depends on how meaningful those samples are. Merely adding arbitrary randomness does not create a reliable approximation to model uncertainty.

Once T predictive samples are available, the computation is simple. Let p_t be the class-probability vector from sample t:

mean_p = mean(p_1, ..., p_T)

predictive_entropy = entropy(mean_p)
expected_entropy = mean(entropy(p_1), ..., entropy(p_T))

bald_score = predictive_entropy - expected_entropy

For a batch with shape [T, N, C], where N is the number of candidate examples and C the number of classes, the same idea can be applied vectorially.

A minimal pseudo-code implementation is:

function entropy(p):
    return -sum(p * log(clamp(p, epsilon, 1)))

function bald(probability_samples):
    mean_probability = mean(probability_samples, axis="samples")
    total = entropy(mean_probability)
    individual = mean(entropy(probability_samples), axis="samples")
    return total - individual

The clamp is a numerical safeguard for log(0). In production, use stable primitives from the numerical framework and confirm which axis represents classes and which represents model samples.

With finite samples and floating-point arithmetic, a theoretically non-negative mutual-information estimate can occasionally become slightly negative from numerical or Monte Carlo error. A tiny negative value should not be interpreted as meaningful negative information.

Use BALD inside an active learning loop

A basic pool-based active learning cycle looks like this:

1. Train on the current labeled set.
2. Produce multiple predictive samples for each unlabeled candidate.
3. Compute a BALD score for each candidate.
4. Select a labeling batch.
5. Obtain labels and move those examples into the labeled set.
6. Retrain or update the model.
7. Repeat until the labeling budget or stopping criterion is reached.

The scoring step is only one part of the system. Several design choices determine whether the resulting labels are actually useful.

Score a manageable candidate pool

Repeated inference can be expensive. If the unlabeled pool contains millions of examples, evaluating many model samples for every item on every round may dominate the active learning cost.

A practical pipeline can first apply inexpensive eligibility rules or sample a candidate subset, then compute BALD only within that subset. This changes the selection space, so the prefilter should not systematically exclude important regions of the deployment distribution.

Select batches, not just isolated points

Taking the top k BALD scores independently can produce a redundant batch. If many near-duplicate inputs all trigger the same area of model disagreement, labeling all of them may provide less information than labeling a more diverse set.

A common system-level pattern is therefore:

high information score + diversity constraint -> labeling batch

The diversity mechanism is separate from BALD. It might use embeddings, clustering, deduplication, domain groups, or another task-appropriate rule. The important point is that BALD estimates informativeness per candidate; it does not by itself guarantee that a batch covers distinct information.

Retrain before assuming scores remain valid

BALD scores depend on the current labeled data and current model uncertainty. Once new labels are incorporated, the model’s beliefs can change. Scores computed several acquisition rounds earlier may no longer represent the same information value.

How often to retrain is an engineering trade-off. Frequent retraining keeps acquisition scores current but costs more compute. Less frequent retraining reduces training cost but can select examples using stale uncertainty estimates.

Do not confuse disagreement with correctness

A high BALD score means the predictive samples disagree. It does not mean the example is mislabeled, out of distribution, representative, or guaranteed to improve a chosen evaluation metric.

Several failure modes matter in practice.

Poor uncertainty approximations produce poor acquisition scores

BALD assumes that variation across predictive samples represents uncertainty about model parameters in a useful way. If an ensemble collapses to nearly identical models, or a stochastic approximation produces variation unrelated to plausible model beliefs, the score loses its intended interpretation.

Before trusting acquisition rankings, inspect the underlying predictive samples. A sophisticated information-theoretic formula cannot repair an uninformative uncertainty model.

Outliers can consume the labeling budget

An unusual input can cause strong model disagreement simply because it is far from the training distribution. Labeling some such examples can be valuable when they represent real deployment traffic. But repeatedly selecting irrelevant corruption, malformed records, or impossible inputs wastes annotation effort.

Eligibility checks and data-quality filters should therefore happen before, or alongside, uncertainty-based acquisition.

Label noise limits what another label can teach

If annotators cannot reliably determine the target from an input, a high-uncertainty example may remain uncertain after labeling. This is especially important when the task definition itself is subjective.

For ambiguous cases, improving annotation guidelines, collecting multiple judgments, or changing the target representation may be more useful than repeatedly acquiring similar examples.

Class imbalance can distort what is operationally useful

A pure BALD ranking optimizes its acquisition score, not business coverage or class quotas. If rare but important cases appear infrequently in the candidate pool, uncertainty sampling alone may not acquire enough of them.

Operational constraints can be layered on top of acquisition, for example by stratifying candidate groups or reserving part of the labeling budget for coverage. Those constraints change the selection objective deliberately rather than pretending that one scalar score captures every requirement.

Evaluate the acquisition strategy, not only the final model

Active learning is useful only if it reaches a desired model quality with fewer or cheaper labels than a reasonable baseline.

A fair experiment should compare acquisition strategies under the same labeling budget. Random sampling is an important baseline because it reveals whether the uncertainty machinery is adding value at all.

A useful evaluation records model quality after each acquisition round:

number of labels -> validation metric

The validation set should remain fixed and should represent the intended deployment task. Do not move actively selected training examples into the validation set, because that makes comparisons across rounds difficult to interpret.

Also measure acquisition cost. If BALD saves 10% of labels but requires an expensive ensemble and many inference passes over a huge pool, the total system may still be less attractive than a simpler strategy. The relevant trade-off is not label count alone; it is label cost, model quality, compute, latency of the acquisition cycle, and operational complexity together.

Know when a simpler strategy is enough

BALD is most compelling when labels are costly, the model can express useful epistemic uncertainty, and unlabeled examples vary substantially in how informative they are.

A simpler strategy may be preferable when labels are cheap, the dataset is small enough to label completely, retraining dominates the project cost, or there is no credible way to obtain multiple model-belief samples. Predictive entropy can also be a reasonable baseline when separating disagreement from shared ambiguity is not important enough to justify repeated inference.

For some tasks, data coverage matters more than uncertainty. If the main problem is missing entire regions of the input space, stratified sampling or diversity-based selection may be easier to reason about. Active learning should solve the actual labeling bottleneck rather than add uncertainty machinery by default.

Conclusion

BALD improves the active learning mental model by asking a more specific question than “which prediction is uncertain?” It asks where plausible model beliefs disagree about the label.

The score expresses that idea as predictive entropy minus expected per-model entropy. High predictive uncertainty with low individual uncertainty indicates disagreement; high uncertainty shared by every model does not.

That distinction can make labeling budgets more informative, but only when the predictive samples represent meaningful model uncertainty. In practice, combine the score with data-quality controls, batch diversity, realistic cost accounting, and a random-sampling baseline. BALD is an acquisition signal, not a guarantee that a selected example will improve the model.

Artificial Intelligence Active Learning Uncertainty Model Evaluation