A classifier can keep seeing familiar inputs and still make worse decisions after deployment. One reason is that the frequency of the classes has changed.

Imagine a model trained to classify support tickets as billing, account, or technical. During training, billing tickets made up 20% of examples. After a pricing migration, billing issues temporarily rise to 50%. The model has not changed, but one part of the environment has: the prior probability of each class.

This pattern is called label shift. Under label shift, class frequencies change while the distribution of inputs within each class stays approximately stable. That assumption is strong, but when it is reasonable, it gives us a useful way to reason about monitoring and even correct predicted probabilities without retraining the classifier.

This article builds that mental model, derives the simplest prior correction, explains what must be true for it to work, and develops a practical workflow for deciding whether label-shift correction is appropriate.

Start with class frequencies, not model scores

Suppose a binary classifier predicts whether an incoming request is urgent or routine.

The training population was:

urgent:  10%
routine: 90%

Later, an incident changes the production population to:

urgent:  30%
routine: 70%

The class prior for urgent has tripled. Even if urgent requests still look like urgent requests and routine requests still look like routine requests, the meaning of the same evidence has changed.

Consider an input whose features are moderately compatible with either class. In the training population, the model learned in a world where routine requests were nine times as common as urgent ones. In the new population, routine requests are only about 2.3 times as common.

The evidence in the input has not necessarily changed. The base rate against which that evidence should be interpreted has.

This is the central mental model for label shift:

same class-conditional evidence
+ different class frequencies
= different posterior probabilities

That distinction matters because many other forms of production drift do not have this structure.

What label shift assumes

Let x represent the input and y the class. We will use source for the population used to train or validate the model and target for the current deployment population.

Label shift assumes that:

P_target(y) != P_source(y)

while approximately preserving:

P_target(x | y) = P_source(x | y)

In plain language, the proportion of each class can change, but examples from a given class should continue to look as they did before.

For the support-ticket example, a billing surge can fit this assumption if billing tickets become more common but the language and characteristics of billing tickets remain broadly similar. It does not fit if a new billing system creates an entirely new type of ticket that was absent from the source data.

The assumption also differs from covariate shift, where the input distribution changes while the relationship between inputs and labels is assumed to remain stable in a different sense. Real production systems can experience several kinds of shift at once, so attaching a drift name to a dashboard change is not enough. The assumed data-generating relationship determines which correction, if any, is defensible.

Why a changed prior changes the posterior

A probabilistic classifier is usually useful because we care about a posterior quantity such as:

P(y | x)

Bayes’ rule says:

P(y | x) = P(x | y) P(y) / P(x)

Under the label-shift assumption, P(x | y) stays the same but P(y) changes. The posterior should therefore change as well.

Suppose the source model provides a calibrated estimate of the source posterior:

P_source(y | x)

Using Bayes’ rule, the target posterior is proportional to:

P_target(y | x)
  ∝ P_source(y | x) * P_target(y) / P_source(y)

For each class, we multiply the source posterior by a prior ratio:

weight(y) = P_target(y) / P_source(y)

Then we normalize the adjusted values so they sum to one.

This is prior-probability correction. It is simple enough to implement in a few lines, but its assumptions are more important than its code.

Work through a small correction

Return to the urgent-request classifier. Suppose its source priors were:

urgent:  0.10
routine: 0.90

and we have reliable evidence that the current target priors are:

urgent:  0.30
routine: 0.70

For one request, the model returns source probabilities:

urgent:  0.40
routine: 0.60

First calculate the prior ratios:

urgent weight  = 0.30 / 0.10 = 3.000
routine weight = 0.70 / 0.90 ≈ 0.778

Apply them to the model probabilities:

urgent:  0.40 * 3.000 = 1.200
routine: 0.60 * 0.778 ≈ 0.467

These are unnormalized scores, not probabilities. Normalize them by their sum:

total ≈ 1.667

urgent:  1.200 / 1.667 ≈ 0.72
routine: 0.467 / 1.667 ≈ 0.28

The same input that received an urgent probability of 0.40 under the source population receives about 0.72 after correcting for the new class priors.

Nothing about this calculation says the model suddenly found stronger urgent features in the request. The adjustment says that the existing evidence should be interpreted in a population where urgent cases are more common.

A compact implementation

The calculation can be expressed independently of any model API:

for each class y:
    adjusted[y] = predicted[y] * target_prior[y] / source_prior[y]

normalizer = sum(adjusted)

for each class y:
    adjusted[y] = adjusted[y] / normalizer

In production code, validate that the priors are non-negative and normalized, handle classes with zero or extremely small source probability deliberately, and perform calculations in a numerically stable way when probabilities or class counts are extreme.

The algorithm is the easy part. Obtaining trustworthy priors and validating the label-shift assumption are the difficult parts.

Correction requires meaningful probability estimates

The derivation treats the model output as an estimate of P_source(y | x). A raw score that merely ranks classes is not automatically such an estimate.

For example, a classifier may output:

urgent:  0.80
routine: 0.20

If examples receiving an 0.80 urgent score are actually urgent only 55% of the time on source-like data, treating 0.80 as a posterior probability can make the prior correction misleading.

Before using probability-based correction, evaluate calibration on held-out source data. Calibration does not prove that label shift holds, and perfect source calibration does not protect against arbitrary production drift. It only supports one requirement of the correction: that the quantities being reweighted behave like useful source posterior probabilities.

If the downstream application needs only a ranking rather than probabilities or threshold decisions, changing the posterior may not even solve the relevant problem. Define the decision you need before adding correction machinery.

Where target priors come from

The correction above assumed that P_target(y) was known. In many deployments, that is the hardest quantity to obtain.

Delayed labels

Some systems eventually receive ground-truth labels. Fraud investigations may finish days later; support tickets may be resolved into known categories; human review may label a sample of model decisions.

When representative target labels are available, estimating class proportions is conceptually straightforward: count the classes over an appropriate window. The practical difficulty is deciding whether the labels are timely and representative enough for the current population.

A convenience sample can be badly biased. If humans review only high-risk predictions, class frequencies in the reviewed set do not directly describe all production traffic.

Unlabeled target data

Methods also exist for estimating target class proportions from model predictions and source validation behavior. A common family of approaches uses a confusion matrix or expected prediction statistics measured on labeled source data, then asks which mixture of source classes could explain the prediction distribution observed on unlabeled target inputs.

The intuition is useful even without implementing a particular estimator. Suppose a classifier has stable source behavior:

true urgent cases  -> predicted urgent 80% of the time
true routine cases -> predicted urgent 10% of the time

If the overall fraction of target examples predicted urgent rises sharply, one possible explanation is that true urgent cases have become more common. Given the source error rates, we can solve for a class mixture that would produce the observed prediction rate.

But this inference depends directly on label shift. If urgent cases have changed so that the classifier now detects only 50% of them, the old confusion behavior is stale and the estimated target prior can be wrong.

Unlabeled estimation therefore does not remove the need to reason about drift. It moves more responsibility onto the assumptions.

A binary mixture example

Suppose source validation tells us:

P(predicted urgent | true urgent)  = 0.80
P(predicted urgent | true routine) = 0.10

Let q be the unknown fraction of urgent cases in the target population. If those class-conditional prediction rates remain stable, the expected target fraction predicted urgent is:

0.80q + 0.10(1 - q)

Now suppose 31% of target inputs are predicted urgent. Then:

0.31 = 0.80q + 0.10(1 - q)
0.31 = 0.70q + 0.10
0.21 = 0.70q
q = 0.30

The implied target prior is 30% urgent.

This teaching example uses hard predictions and a two-class system because the arithmetic is transparent. Real estimators may use soft prediction probabilities and multiple classes, and they need safeguards for sampling error and poorly conditioned systems. Do not treat this small equation as a complete production estimator.

The example does reveal the core dependency: if different true classes produce sufficiently distinguishable prediction patterns, changes in the observed mixture can contain information about changes in class proportions.

Identifiability can fail

Not every classifier provides enough information to estimate label proportions reliably.

Imagine a binary classifier whose source behavior is:

P(predicted urgent | true urgent)  = 0.51
P(predicted urgent | true routine) = 0.50

Its predictions look almost the same for both true classes. A change in the overall predicted-urgent rate tells us very little about how the target class mixture changed.

In multiclass settings, the analogous problem appears when the class-conditional prediction patterns are not sufficiently independent. Solving for target priors can then become unstable: small measurement errors in observed prediction rates can produce large changes in the estimated priors.

This is a boundary condition worth checking before building automated correction. A mathematically solvable system can still be too ill-conditioned to trust with finite, noisy data.

Detecting drift is not the same as diagnosing label shift

A monitoring system might notice that:

  • predicted class frequencies changed;
  • average confidence changed;
  • input embeddings moved;
  • selected feature distributions changed;
  • downstream error rates increased.

These are useful signals, but none by itself proves label shift.

Predicted class frequencies can change because true class frequencies changed, because inputs within a class changed, because a data pipeline changed, or because the relationship between inputs and labels changed. An embedding-distance alert can likewise detect a population change without telling you which probabilistic assumption failed.

Treat label shift as a hypothesis to investigate, not a label to assign automatically to any drift alert.

When labels arrive eventually, compare class-conditional behavior across source and target data. Ask questions such as:

Did class proportions change?
Did accuracy or recall within each class change?
Did important features change within the same class?
Did score distributions change within the same class?

If class proportions moved while within-class behavior remained reasonably stable, label shift becomes a more plausible approximation. If within-class behavior moved substantially, prior correction alone is unlikely to describe the problem.

Evaluate the correction as a decision system

A corrected probability vector is not useful merely because it follows a derivation. Evaluate whether it improves the decisions your application actually makes.

Start with a time period or dataset where target labels are available. Compare at least:

uncorrected model
corrected model
simple baseline, when relevant

Then measure metrics aligned with the use case. For probabilistic predictions, calibration and proper scoring rules can reveal whether probability quality improved. For thresholded workflows, measure precision, recall, false-positive cost, false-negative cost, or other operational outcomes at the actual decision thresholds.

Also evaluate slices. A global correction can improve an average metric while hurting a region, language, customer segment, or rare class that matters operationally.

Most importantly, evaluate on data from the target regime rather than applying a new prior and reporting performance on the unchanged source validation set. The entire purpose of the correction is to adapt to a different class mixture.

Watch the time window

Production priors are often not constant. An incident can cause a two-hour spike; weekday traffic can differ from weekend traffic; a seasonal event can change class frequencies for weeks.

A very long estimation window can react too slowly. A very short window can produce noisy prior estimates, especially for rare classes.

This creates a bias-variance trade-off:

long window  -> smoother estimate, slower response
short window -> faster response, more sampling noise

Choose the window from the rate at which the population can change and the amount of traffic available. If the application is high impact, consider confidence intervals, minimum sample requirements, or human review before allowing a large automatic adjustment.

Do not confuse responsiveness with correctness. Updating priors every minute does not help if the estimator has too little data to identify a meaningful change.

Common mistakes

Reweighting because predicted frequencies changed

A change in predictions is evidence that something changed, not proof that class priors changed while P(x | y) stayed fixed. Diagnose the shift before applying a correction whose assumptions may not hold.

Using training-set priors blindly

The source prior in the correction should describe the population represented by the model probabilities. Dataset construction can distort this. If training used deliberate class balancing, oversampling, or case-control sampling, the raw class counts in the training file may not represent the prior implicit in a calibrated deployment model.

Document how the model was trained and calibrated before choosing the denominator in the prior ratio.

Correcting uncalibrated scores as if they were probabilities

Softmax output is normalized, but normalization alone does not guarantee empirical calibration. If probability quality matters, validate it rather than inferring it from the output format.

Ignoring new classes

Prior correction can only redistribute probability among classes the model already represents. If deployment introduces a genuinely new class, increasing and decreasing old class priors cannot create a model of that new class.

An unknown or emerging class is a different problem and may require data collection, model changes, abstention, or a broader detection strategy.

Letting extreme ratios dominate silently

If a source class is extremely rare, dividing by its tiny source prior can produce a very large weight. The arithmetic may be valid under ideal assumptions while the estimate is statistically fragile in practice.

Investigate rare classes separately, require adequate sample support, and test sensitivity to plausible prior estimates before allowing large automatic changes.

When label-shift correction is a good fit

Prior correction is worth considering when class frequencies plausibly change faster than the relationship between class identity and inputs, the classifier provides meaningful source probabilities, and target priors can be estimated with adequate evidence.

Examples can include temporary changes in ticket categories, disease prevalence changes for a fixed diagnostic process, or changes in the mix of known document types. Whether any specific deployment actually satisfies label shift must be tested rather than assumed from the domain name.

A simpler response is often better when you already have fresh representative labeled data and can retrain or recalibrate the model reliably. Retraining can adapt to changes that go beyond class priors, whereas label-shift correction intentionally addresses a narrower problem.

Correction is a poor fit when new classes appear, class-conditional inputs change substantially, labels are so biased that target priors cannot be estimated, or the prediction system has changed for reasons unrelated to population composition.

Conclusion

Label shift separates one specific production change from the broad idea of model drift: class frequencies change while the distribution of inputs within each class remains approximately stable. Under that assumption, a source posterior can be adjusted by the ratio of target to source class priors and then renormalized.

The formula is not the hard part. The hard part is establishing that the assumptions are reasonable, obtaining trustworthy target priors, and evaluating the corrected system on the decisions that matter.

Use the mental model in that order: detect a change, diagnose whether label shift is a plausible explanation, estimate the new priors, apply correction only when its inputs are defensible, and validate the result on target data. A narrow correction with tested assumptions is more useful than a sophisticated drift response applied to the wrong problem.

Artificial Intelligence Machine Learning Model Evaluation Model Monitoring