A classifier usually returns one label or a vector of scores. That is convenient when the application must choose one answer, but it hides an important distinction: some inputs strongly support one class, while others leave several classes plausible.

Conformal prediction provides a way to expose that ambiguity. For classification, it can return a prediction set containing one or more labels instead of forcing every input into a single choice. With an appropriate calibration procedure and statistical assumptions, the method can target a long-run coverage level such as 90%: roughly speaking, the true label should appear in the prediction set for at least that proportion of future examples.

The guarantee is useful, but it is easy to overstate. It is a marginal coverage guarantee over examples drawn under the same exchangeability assumption as the calibration data, not a promise that every individual prediction set is 90% likely to be correct. This article builds the method from a small example, explains the finite-sample threshold, and shows how to evaluate whether the resulting sets are useful in practice.

Start with the decision you actually need

Suppose a support classifier routes tickets into three queues:

billing
account
technical

For one ticket, the model produces:

billing:   0.82
account:   0.11
technical: 0.07

For another:

billing:   0.41
account:   0.36
technical: 0.23

If you take only the largest score, both tickets become billing. Yet the second decision is much less clear.

A prediction-set system can express the difference directly:

ticket A -> {billing}
ticket B -> {billing, account}

The first set says one class is sufficient under the chosen conformal rule. The second says the system needs to retain more than one plausible class to meet its coverage target.

This changes the downstream question from:

Which class has the highest score?

into:

Which classes must I keep so that the prediction procedure reaches its target coverage on comparable future data?

That is especially useful when a downstream workflow can handle ambiguity, for example by showing several candidate diagnoses to an expert, searching several intent-specific indexes, or sending uncertain cases to a broader review queue.

The mental model: calibrate how surprising the true label looks

Split conformal prediction uses three distinct data roles:

training data    -> fit the classifier
calibration data -> measure prediction errors
future input     -> build a prediction set

The calibration set is the key. For each calibration example, the classifier already knows nothing about its label during fitting because these examples were held out from training. You compare the model’s scores with the known true label and calculate a nonconformity score: a number where larger values mean the true label looked less compatible with the model’s prediction.

A simple classification score is:

s(x, y) = 1 - p(y | x)

where p(y | x) is the classifier’s score for the true class. If the true class receives probability 0.90, its nonconformity score is 0.10. If it receives 0.25, the score is 0.75.

The calibration scores therefore answer a concrete question:

How bad can the model’s score for the true label get on held-out examples before the case becomes unusually difficult?

Conformal prediction chooses a threshold from those observed scores. At inference time, a candidate label is included when its nonconformity score is no larger than that threshold.

Work through a small calibration example

Assume the classifier was already trained. Keep five separate labeled examples for calibration and record the probability assigned to each example’s true class:

example 1: 0.92
example 2: 0.81
example 3: 0.74
example 4: 0.63
example 5: 0.40

Using 1 - p(true class) gives these nonconformity scores:

0.08, 0.19, 0.26, 0.37, 0.60

Suppose the target miscoverage rate is:

alpha = 0.20

so the desired coverage is 1 - alpha = 0.80, or 80%.

For split conformal prediction with n calibration examples, a standard finite-sample threshold uses the order statistic at rank:

k = ceil((n + 1) * (1 - alpha))

For n = 5:

k = ceil(6 * 0.80)
  = ceil(4.8)
  = 5

The fifth-smallest calibration score is 0.60, so the threshold is:

q = 0.60

For a new input, include every class y satisfying:

1 - p(y | x) <= 0.60

which is equivalent in this example to:

p(y | x) >= 0.40

Now consider a new ticket with:

billing:   0.52
account:   0.41
technical: 0.07

Its candidate nonconformity scores are:

billing:   0.48
account:   0.59
technical: 0.93

The prediction set is therefore:

{billing, account}

This example is intentionally tiny so the arithmetic is visible. A five-example calibration set produces a coarse threshold and is not a sensible production calibration budget.

The finite-sample correction matters

It may be tempting to take an ordinary empirical 80th percentile of the five calibration scores. Split conformal prediction instead uses the (n + 1) correction in the rank.

That correction is part of the finite-sample coverage argument. Under exchangeability, the calibration nonconformity scores and the new example’s true-label score can be treated symmetrically. The new score’s rank among the n + 1 scores is what drives the coverage result.

For a desired miscoverage rate alpha, the rank

ceil((n + 1) * (1 - alpha))

is capped by the available calibration order statistics when the requested quantile is representable. With very small calibration sets and very ambitious coverage targets, the finite-sample resolution becomes a real limitation. You cannot obtain arbitrarily fine error control from only a handful of calibration examples.

Libraries differ in how they expose quantile calculations and edge cases, so verify their conformal-specific convention rather than substituting a generic percentile call without checking its definition.

Coverage is not the same as confidence

The central split-conformal statement is about repeated examples. Under the required assumptions, a prediction set C(X) is constructed so that its marginal coverage satisfies a bound of the form:

P(Y in C(X)) >= 1 - alpha

This does not mean that after observing a particular input and its set, you can generally say:

P(true label is in this particular set | this input) = 0.90

Those are different claims. Standard split conformal prediction controls coverage averaged over future examples from the relevant distribution. It does not automatically provide exact conditional coverage for every subgroup, input region, or individual case.

That distinction has practical consequences. A system can achieve 90% overall coverage while performing worse for a rare class or a subgroup that is poorly represented in calibration data. Always inspect coverage at slices that matter to the application, even though small slices will have noisier estimates.

Exchangeability is the assumption behind the guarantee

The usual conformal guarantee relies on calibration and future examples being exchangeable. Informally, their ordering should not carry information about which examples are easier or harder for the prediction rule.

Independent and identically distributed examples are a common case where exchangeability holds. Real systems can violate the assumption through distribution shift, temporal drift, changing class frequencies, new devices, new languages, or changes in data collection.

Consider a ticket classifier calibrated on last year’s product. If a new product launch creates a new pattern of technical requests, the old calibration-score distribution may no longer represent current inputs. The conformal procedure will still produce sets, but the historical coverage guarantee should not be assumed to transfer unchanged to the shifted population.

This is why conformal prediction does not remove the need for monitoring. Track realized coverage when labels eventually arrive, inspect score distributions, and recalibrate when the deployment population changes materially.

For strongly time-dependent or otherwise non-exchangeable data, specialized conformal methods may be more appropriate. The simple split procedure in this article should not be presented as a universal solution to arbitrary distribution shift.

Prediction-set size tells you about usefulness

Coverage alone is not enough to evaluate a conformal classifier. A trivial system that returns every possible class has 100% coverage but may be useless.

You therefore need to measure set efficiency, commonly through prediction-set size.

For a three-class task:

{billing}                     -> size 1
{billing, account}            -> size 2
{billing, account, technical} -> size 3

Two conformal systems can both achieve the target coverage while one produces much smaller sets. Smaller sets are generally more actionable, provided coverage is preserved on the population that matters.

Useful evaluation summaries include:

  • empirical coverage on untouched test data;
  • average or median prediction-set size;
  • the distribution of set sizes, not only its mean;
  • coverage and set size by important class or operational slice;
  • the fraction of empty sets, if the chosen scoring rule can produce them;
  • downstream cost, such as how often a human must review multiple candidates.

The goal is not to minimize set size at any cost. It is to obtain useful sets while meeting the required coverage behavior.

The nonconformity score changes the sets you get

The simple score

1 - p(y | x)

is easy to teach and implement, but it is not the only conformal classification score.

Suppose a ten-class model often spreads probability across several plausible labels. A fixed probability threshold derived from 1 - p(y | x) can produce sets whose sizes vary substantially. Other conformal methods rank or accumulate class probabilities before defining the nonconformity score, which can improve efficiency for some model and data distributions.

The important separation is:

base model        -> produces class scores
nonconformity rule -> converts scores into calibration quantities
conformal quantile -> sets the coverage-oriented threshold

Changing the nonconformity rule can change set efficiency even when the base classifier is unchanged. It does not remove the need for a held-out calibration set or the assumptions behind the coverage guarantee.

Start with a simple score when it meets the application’s needs. More elaborate scoring is justified when evaluation shows that prediction sets are too large or otherwise poorly shaped for the task.

Calibration data must remain separate from model fitting

Split conformal prediction is called “split” because model training and conformal calibration use separate examples.

A clean workflow is:

training split
    -> fit model and model hyperparameters

calibration split
    -> freeze model
    -> compute nonconformity scores
    -> choose conformal threshold

test split
    -> estimate coverage and set efficiency

Do not train the classifier on the calibration examples and then treat their in-sample scores as if they came from untouched data. Training examples often look easier to the fitted model, which can make the conformal threshold too optimistic.

Likewise, avoid repeatedly choosing scoring rules, calibration strategies, and operational thresholds by looking at the final test set. Once test results influence design choices, that set is no longer an untouched estimate of final behavior.

The cost of split conformal prediction is therefore partly statistical: calibration consumes labeled examples that otherwise could have been used for fitting the base model. When labeled data are scarce, methods that reuse data more efficiently can be attractive, but they also add complexity. A simple split is often the easiest implementation to reason about and audit.

Model quality still matters

Conformal prediction can wrap a weak classifier and still target marginal coverage by returning larger sets. It does not make the underlying model more discriminative.

Imagine two classifiers on the same ten-class problem. Both are conformalized to 90% target coverage. A strong classifier may often produce singleton sets:

{technical}

A weak classifier may need sets such as:

{technical, billing, account, sales, other}

Both can have similar coverage, but their usefulness is very different.

This creates a productive division of responsibilities:

base model quality -> how sharply useful labels can be separated
conformal layer    -> how the prediction rule controls coverage

Improving the classifier, features, or training data can make conformal sets smaller. Conformal calibration should not be used to hide poor predictive performance.

Common mistakes break the intended interpretation

Calling the target coverage a per-example probability

A 90% marginal coverage target is a property of the prediction procedure over examples. It is not automatically a calibrated 90% probability statement about each individual set.

Reusing training data for calibration

If the same examples shaped the fitted model, their scores do not play the clean held-out role assumed by ordinary split conformal prediction. Keep a separate calibration split unless you deliberately use a method designed for data reuse.

Ignoring distribution shift

Coverage depends on the relationship between calibration and future data. A threshold calibrated before a major population change may no longer deliver the expected operational coverage.

Evaluating only coverage

Returning every label can achieve excellent coverage. Always measure set size and downstream usefulness alongside coverage.

Treating model probabilities as a conformal guarantee

Probability calibration and conformal coverage are related to uncertainty but are different properties. A well-calibrated classifier does not automatically provide finite-sample conformal coverage, and conformal prediction does not automatically make its base probabilities calibrated.

Tuning on the test set

If you choose the conformal score or other design decisions after inspecting final test behavior, the test set becomes part of development. Reserve untouched data for the final evaluation when you need an honest estimate.

When prediction sets are a good fit

Split conformal classification is useful when the application can act on a set of plausible labels and a measurable coverage target is more useful than an unsupported confidence heuristic.

Good examples include candidate generation for human review, medical decision support where an expert considers several possibilities, intent routing that can search multiple specialized paths, and classification systems that abstain or escalate when sets become large.

A prediction set is less useful when the downstream system must choose exactly one action and has no meaningful way to handle alternatives. In that case, a carefully evaluated top-1 classifier, calibrated probabilities, cost-sensitive decision rule, or abstention policy may match the operational problem more directly.

Conformal prediction is also not a substitute for handling severe distribution shift. If future inputs are unlike the calibration population, first address how the system detects, models, or adapts to that change.

Conclusion

Split conformal classification adds a small but important layer around an existing classifier. Hold out calibration examples, measure how nonconforming their true labels look, choose the finite-sample conformal threshold, and include new candidate labels that fall within that threshold.

The practical value comes from combining two measurements: coverage tells you how often the true label is retained, while set size tells you whether retaining it is operationally useful. Keep the calibration data independent of model fitting, interpret the guarantee as marginal rather than per-example, and re-evaluate when the deployment distribution changes. With those boundaries clear, prediction sets provide a disciplined way to represent classification ambiguity without pretending that every input deserves one certain answer.

Artificial Intelligence Machine Learning Model Evaluation Uncertainty