A classifier that emits 0.93 for one class does not automatically provide a statistical statement that the class is correct with probability 0.93. Conformal prediction takes a different route: it uses held-out labeled examples to construct a set of candidate labels with a target marginal coverage level.
For split conformal classification, the base model can remain fixed. The guarantee comes from ranking a test example’s conformity or nonconformity score against scores computed on an exchangeable calibration sample. That assumption is the part that gives the coverage statement its scope.
A score turns model output into a rank
Consider a fixed classifier with class probabilities p(y | x). A simple nonconformity score for a labeled example is
s(x, y) = 1 - p(y | x)A high probability for the observed label produces a small score. Other score definitions are possible, including scores that use cumulative probability mass across ranked classes. The conformal mechanism does not require the score to be a calibrated probability.
With split conformal prediction, model fitting and calibration are separate. The model is fitted before the calibration scores are computed. A quantile of those held-out scores then defines a threshold for a new input.
For a candidate label y, the system evaluates s(x_new, y) and includes the label when its score falls within the calibrated threshold. The output can contain one label, several labels, or, depending on the score and threshold, every label.
This is set-valued prediction rather than a new point classifier.
Coverage is marginal across test examples
For exchangeable calibration and test examples, standard conformal constructions provide a finite-sample marginal coverage statement. At a target error level alpha, the prediction set contains the true label with probability at least approximately 1 - alpha, with the exact finite-sample form determined by the conformal quantile convention.
The word marginal sets an important boundary. The guarantee averages over a fresh test example drawn under the same exchangeable setup. It does not state that every individual input has conditional coverage of 1 - alpha.
An input from a difficult region can have a different error rate from an input in an easy region while the overall marginal target is still met. The same issue can appear across classes or other subgroups.
This makes aggregate coverage necessary but not sufficient for diagnosing set behavior. Per-class coverage, subgroup coverage, and set-size distributions can expose structure that one overall coverage number hides.
Exchangeability carries the guarantee
Exchangeability means, informally, that the joint distribution is unchanged when the relevant examples are reordered. Independent and identically distributed examples satisfy this condition, but exchangeability is the broader concept used by standard conformal arguments.
The rank of a new score among calibration and test scores has the required symmetry only under the stated data assumptions. If deployment data shifts after calibration, that symmetry can fail.
A change in class prevalence, sensor conditions, document source, geography, or another input-generating process can alter score distributions. The stored conformal threshold does not detect that change by itself.
There are conformal variants for specific forms of shift, but they require additional assumptions or weighting mechanisms. A guarantee from the exchangeable setting should not be carried over to shifted data without checking that the new method’s conditions apply.
The calibration split has two jobs
The calibration data determines the score threshold, so it needs labels and must not be reused casually.
First, it supplies the empirical score distribution used for the conformal rank. Second, its separation from model fitting keeps the scoring function fixed with respect to the calibration examples in the usual split-conformal construction.
If model parameters are adjusted after examining calibration outcomes, the original calibration scores no longer describe a fixed scoring rule in the same way. A fresh calibration split is the clean option after such model changes.
Calibration size also affects threshold granularity. With only a small number of calibration examples, there are only a small number of possible empirical ranks. Requesting a very small error rate can therefore produce conservative or uninformative sets because the finite sample cannot resolve arbitrary tail probabilities.
The nominal coverage target is not a substitute for enough representative calibration data.
Set size measures usefulness, not validity
A prediction set that contains every possible label can achieve high coverage while providing little discrimination. Coverage alone therefore does not describe whether the sets are useful.
Set size is a natural companion measurement in classification. For the same coverage target, a stronger base model or a score better aligned with the task can often concentrate candidate sets more tightly. That is an efficiency property of the prediction sets, separate from the validity of the marginal coverage statement.
Comparisons should hold the coverage target and evaluation population fixed. Reporting smaller sets after silently accepting lower empirical coverage mixes two different effects.
The full distribution of set sizes can also matter. An average of 1.4 labels can arise from mostly singleton sets with a few large sets, or from a different mixture. Those patterns can imply different behavior for downstream systems that route large sets to review.
Probability calibration and conformal coverage are distinct
Probability calibration asks whether predicted probabilities correspond to observed event frequencies. Conformal coverage asks whether a constructed set contains the true outcome at the specified marginal rate under its assumptions.
A classifier can have imperfect probability calibration and still serve as the scoring model inside a valid split-conformal procedure. Conversely, calibrated class probabilities do not by themselves create the finite-sample set-coverage guarantee associated with conformal prediction.
The two ideas can interact. Probability estimates can be inputs to a conformal score, and calibration quality can affect set efficiency. Their guarantees remain different, so evaluation should keep the terms separate.
This distinction is especially useful in APIs. A field named confidence suggests a scalar probability-like quantity. A conformal result is more naturally represented as a set plus metadata describing the target coverage, calibration version, score definition, and population assumptions.
Model updates invalidate stored scores
A conformal threshold belongs to a particular scoring function and calibration sample. Changing model weights can change the score assigned to every example. Changing preprocessing, tokenization, label definitions, or score construction can do the same.
Reusing the old threshold after such a change disconnects it from the score distribution that produced it.
Versioning the base model and conformal calibrator together makes this dependency explicit. The same principle applies to data changes: if the operating population no longer resembles the calibration population closely enough for the required assumptions, recalculating the threshold on representative data is more meaningful than treating the old coverage target as permanent.
Conformal prediction is attractive partly because it can sit around an existing classifier without changing that classifier’s weights. The wrapper is still statistical state. Its guarantee is attached to the score, calibration data, and exchangeability conditions as a unit; separating the threshold from those conditions turns a precise coverage statement into an unsupported number.