Regularize Classifier Targets with Label Smoothing

A classifier trained with one-hot targets is rewarded for pushing the target class probability toward one and every other class probability toward zero. Cross-entropy keeps applying pressure in that direction even after the predicted class is already correct. Label smoothing changes that pressure by replacing the exact one-hot target with a distribution that reserves some mass for other classes.

That small change affects more than the target tensor. It changes the gradient on every output logit, limits the incentive for extreme class separation, and alters how predicted probabilities should be interpreted.

Soft targets change the cross-entropy objective

Consider a classifier with K classes and smoothing parameter epsilon. A common label-smoothing convention defines the target distribution as:

q_k = (1 - epsilon) * 1[k = y] + epsilon / K

where y is the labeled class. The target class therefore receives:

1 - epsilon + epsilon / K

and each other class receives epsilon / K.

Some libraries or papers distribute the smoothing mass only across non-target classes instead. Under that convention, each non-target class receives epsilon / (K - 1). These definitions are close in spirit but not numerically identical, so an implementation should state which convention it uses.

With predicted probabilities p_k, cross-entropy becomes:

L = -sum_k q_k log(p_k)

The loss still compares a predicted categorical distribution with a target distribution. The difference is that the target no longer sits at a vertex of the probability simplex.

The logit gradient no longer targets exact certainty

For softmax logits z_k, the derivative of cross-entropy with respect to a logit has the familiar form:

dL / dz_k = p_k - q_k

With one-hot targets, the gradient for the labeled class is p_y - 1. It remains negative whenever p_y is below one, so gradient descent continues increasing the target logit relative to the alternatives.

With label smoothing, the target value is below one. Once p_y rises above the smoothed target probability, that component of the gradient changes sign. Non-target classes also have positive target mass, so driving their probabilities arbitrarily close to zero is no longer the objective.

This does not impose a hard cap on a model’s output probability for every input. Parameters are shared across examples, optimization is approximate, and the network is not independently fitting one unconstrained probability vector per sample. The direct statement is narrower: the training target and resulting gradient no longer request a one-hot distribution for each labeled example.

Class margins receive less pressure to expand

Softmax probabilities depend on differences between logits. For the target probability to approach one under ordinary one-hot cross-entropy, the target logit must become increasingly larger than competing logits. Correct classification alone does not end that pressure.

Label smoothing reduces the incentive to keep expanding those margins. This can matter in high-capacity classifiers that can fit training labels with very large logit differences. The regularizer acts at the output objective rather than by directly constraining parameter norms.

That distinction separates label smoothing from weight decay. Weight decay penalizes or shrinks parameters according to the optimizer’s formulation. Label smoothing instead changes the desired output distribution used to compute the data loss. The two mechanisms can coexist because they act on different parts of optimization.

The smoothing distribution encodes an assumption

Uniform smoothing treats every class as an equally plausible recipient of the reserved probability mass. That is a simple regularization choice, not a statement that all classification errors have equal semantic meaning.

For a ten-class problem using the epsilon / K convention with epsilon = 0.1, the target class receives 0.91 and each class receives at least 0.01. The objective therefore contains a uniform component regardless of relations among classes.

This assumption can be awkward when the label space has structure. Confusing two visually similar categories can be materially different from confusing unrelated categories, yet uniform smoothing assigns both alternatives the same target mass. A non-uniform soft target can encode class relations, but then the method is no longer plain uniform label smoothing and the source of that target distribution becomes part of the model design.

Noisy labels and smoothing are separate issues

Label smoothing is sometimes associated with resistance to mislabeled examples because it reduces the one-hot pressure applied by every training label. That does not make it a general correction for label noise.

If an example carries the wrong class, smoothing still assigns the largest target probability to that wrong class. The objective has merely reduced the strength of the target. Systematic annotation errors, class-dependent noise, and ambiguous labeling can require data checks or noise-aware methods rather than a fixed smoothing constant.

The distinction matters when selecting epsilon. A larger value does not estimate the noise rate unless the modeling setup explicitly establishes such a relationship. In ordinary label smoothing, epsilon is a regularization parameter controlling target softness.

Calibration needs separate measurement

Because label smoothing discourages extreme output distributions during training, it can change confidence behavior. That effect should not be treated as a calibration guarantee.

Calibration asks whether predicted probabilities correspond to observed outcome frequencies under a defined evaluation setup. Label smoothing optimizes softened cross-entropy targets. Those are different objectives, and a model can still be miscalibrated after training with smoothed labels.

This is especially relevant when probabilities feed a downstream threshold, abstention rule, ranking stage, or cost-sensitive decision. Classification accuracy alone does not show whether those probabilities have the required interpretation. Calibration metrics and reliability analysis should be computed on held-out data appropriate to the deployment distribution.

Post-training calibration methods also solve a different problem. For example, temperature scaling adjusts logits using held-out calibration data while leaving the predicted class unchanged for positive temperatures. Label smoothing modifies training itself and can change the fitted representation and decision boundaries. Treating the two as interchangeable hides that difference.

Smoothing strength interacts with the number of classes

The numeric target assigned to each class depends on both epsilon and K. Under the epsilon / K convention, each class receives a uniform contribution of epsilon / K. As the class count grows, the mass assigned to each individual non-target class becomes smaller even when epsilon stays fixed.

The target class still loses approximately epsilon mass relative to a one-hot target, but the distribution of that mass across alternatives changes with the label-space size. Comparing smoothing constants across tasks without considering the convention and class count can therefore be misleading.

The same issue appears when code moves between frameworks. A parameter named label_smoothing=0.1 is only fully specified once the target construction is known. Tests that inspect the resulting target distribution or loss on a small fixed logit vector can expose convention differences before they affect a larger training run.

Hard labels still matter at evaluation time

Training against softened targets does not require evaluation labels to become soft. If the task defines one correct class per example, accuracy and other hard-label metrics can continue using that definition.

The training target is an optimization device. It should not silently redefine the application contract. A system that needs probability estimates, top-k predictions, selective classification, or cost-weighted decisions still needs evaluation aligned with those outputs.

Label smoothing is most precise when described as a modification to the classification objective: it replaces exact one-hot targets with distributions that reserve probability mass for alternatives. Its useful effects and its limits both follow from that change. It can reduce pressure for extreme logits, but it does not establish calibrated probabilities, repair arbitrary annotation errors, or encode class similarity unless the target distribution is explicitly designed to do so.