A classifier normally returns one label or a vector of class scores. Neither output directly states how many labels should remain plausible when the system needs a controlled error rate. Split conformal prediction adds a calibration layer that turns those scores into prediction sets.
The useful property is not that every individual set has a fixed probability of containing the correct label. Under the standard exchangeability assumption, split conformal methods target marginal coverage across new examples. That distinction shapes both implementation and interpretation.
Calibration turns scores into a threshold
Assume a classifier produces probabilities p(y | x) over a finite label set. A simple nonconformity score for an example with true label y is:
s(x, y) = 1 - p(y | x)Large values indicate that the classifier assigned little probability to the observed label. After the base classifier is fixed, compute this score on a separate calibration set of n labeled examples.
For a target miscoverage rate alpha, a common split conformal construction uses the empirical quantile at rank:
ceil((n + 1) * (1 - alpha))with the rank capped at n when needed. Let the resulting calibration threshold be q. For a new input x, include label y when:
s(x, y) <= qWith the score above, this is equivalent to retaining classes whose predicted probability is at least 1 - q.
The finite-sample correction involving n + 1 is part of the coverage construction. Replacing it with an arbitrary percentile can change the formal guarantee, especially when the calibration set is small.
The guarantee is marginal, not per input
The standard split conformal statement concerns a fresh test pair and the random calibration sample under exchangeability. In simplified form, the resulting set C(X) satisfies:
P(Y in C(X)) >= 1 - alphasubject to the conformal construction and its assumptions.
This probability averages over inputs. It does not imply that every region of the input space has coverage 1 - alpha, nor does it assign that probability to each realized prediction set.
Suppose a system targets 90% marginal coverage. One input segment can have coverage below that level while another segment compensates with coverage above it. Aggregate coverage can still meet the target. Applications that require guarantees within classes or operational groups need a construction designed for that condition rather than an interpretation added after calibration.
This boundary is easy to miss when prediction sets are displayed as uncertainty estimates. A set with three labels is not, by itself, a statement that the correct label has a particular conditional probability of belonging to those three labels.
Exchangeability carries the statistical burden
Split conformal prediction is often described as distribution-free because it does not require a parametric form for the data distribution. It still relies on symmetry assumptions. The calibration examples and the future example must be exchangeable for the usual finite-sample marginal coverage result.
Independent and identically distributed sampling is a familiar sufficient setting, but exchangeability is the more direct condition used by the conformal argument.
Deployment drift can break the practical connection between calibration and future inputs. If the calibration set contains one traffic distribution and production later receives another, the rank of the new nonconformity score among calibration scores no longer has the same symmetry.
The conformal layer does not detect this shift. It continues to apply the stored threshold. Coverage monitoring therefore belongs beside set-size monitoring when the input population can move.
Score design changes set size
Coverage does not determine how informative the sets are. Two valid conformal constructions can target the same marginal coverage and produce different average set sizes.
The score 1 - p(y | x) is simple, but it treats each candidate label through its individual probability. Other classification scores use cumulative probability mass or randomized adjustments to obtain different set behavior. These choices alter which labels enter the set at a given calibration threshold.
The base classifier also matters. If it places high scores on the true class and low scores on alternatives, calibration scores tend to support smaller sets. If several classes receive similar scores, meeting the same coverage target can require larger sets.
Conformal calibration does not repair class discrimination. It wraps the score function with a threshold selected to control a coverage criterion. Better ranking can make the resulting sets more compact, but compactness and coverage remain separate properties.
A calibration split has a distinct role
The calibration set is not ordinary validation data once its scores define the conformal threshold. Choosing model variants, score functions, or hyperparameters after inspecting their conformal performance on that same set can make the final procedure depend on data that the coverage argument treats as calibration observations.
A clean split keeps model fitting and selection upstream, then freezes those choices before conformal calibration. A final test set can remain untouched for reporting empirical coverage and set-size behavior.
The cost is data usage. Holding examples out for calibration leaves fewer examples for fitting the base model. More calibration examples, however, provide finer quantile resolution and reduce the discreteness imposed by a small n.
That resolution is visible directly in the rank formula. The threshold can only move among observed calibration scores, so a small calibration sample supports a limited collection of attainable operating points.
Empty and large sets are meaningful outputs
Depending on the score and threshold, a conformal classifier can return one label, several labels, every label, or in some constructions an empty set. Application code should treat set cardinality as part of the output contract rather than assuming a single prediction.
Large sets often indicate that the score function cannot separate plausible labels strongly enough at the requested coverage level. They can also arise when alpha is small, since stricter coverage generally requires a threshold that admits more labels.
An empty set has a different meaning: no candidate passed the calibrated inclusion rule. Whether empty sets are possible depends on the score construction and any additional conventions applied by the implementation.
Forcing every output to contain exactly one label changes the prediction-set procedure. The resulting single-label interface may be operationally convenient, but its error behavior is no longer established merely by the conformal coverage result for the original sets.
Coverage and set size should be evaluated together
Empirical coverage on untouched data checks how often the true label appears in the returned set. Set size describes how much ambiguity the system exposes while achieving that coverage.
Reporting only coverage can hide a trivial but unhelpful solution that returns nearly every label. Reporting only set size can hide undercoverage. The pair gives a clearer view of the operating point.
Useful diagnostics can also stratify both quantities by class or relevant input segment. Those slices do not transform a marginal guarantee into a conditional one, but they can reveal concentrations of errors or unusually large sets that the aggregate values conceal.
When comparing conformal variants, the target coverage, calibration protocol, base model, and evaluation population need to remain explicit. A smaller set is useful only if the associated coverage criterion is still satisfied under the comparison setup.
Recalibration is tied to the scoring pipeline
The stored conformal threshold belongs to the exact scoring pipeline used to create calibration scores. Replacing the classifier checkpoint, changing preprocessing, modifying label definitions, or changing the nonconformity score can alter the score distribution.
Keeping the old threshold after such a change is not equivalent to calibrating the new pipeline. The new system needs calibration scores produced by its own frozen components.
This makes conformal state small but semantically specific: a threshold may be only one number, yet its validity depends on the model, score definition, calibration population, target error rate, and data-splitting protocol that produced it.
For developers, that boundary is the main implementation value of split conformal prediction. It creates a clear interface between model scores and a coverage-controlled set output, while leaving the assumptions behind that control visible enough to test and monitor.