A classifier trained with one-hot targets receives a strong signal to push the target class probability toward one and every other class probability toward zero. Cross-entropy supports that behavior even after the predicted class is already correct: making the target probability more extreme can still reduce the loss.
Label smoothing changes the target distribution before cross-entropy is computed. Instead of assigning all target mass to one class, it reserves a small amount for the remaining classes. This alters the gradient applied to the logits and reduces pressure toward extreme output distributions.
The mechanism is simple, but its effect depends on the smoothing convention, the task, and what downstream code expects probabilities to mean.
The target distribution controls the logit gradient
For a classifier with logits z and softmax probabilities p, cross-entropy against a target distribution q can be written as:
L = -sum_i q_i log(p_i)The derivative with respect to each logit has a compact form:
dL/dz_i = p_i - q_iWith a one-hot target for class y, q_y = 1 and all other target entries are zero. The target-class gradient remains negative while p_y < 1, so gradient descent keeps increasing its logit relative to the others.
Label smoothing replaces that one-hot vector with a distribution whose entries are less extreme. One common convention for K classes and smoothing coefficient epsilon is:
q_y = 1 - epsilon + epsilon / K
q_i = epsilon / K for i != yUnder this convention, some of the smoothing mass is also assigned back to the target class. Another convention sets q_y = 1 - epsilon and distributes epsilon only among the K - 1 non-target classes. These definitions are close but not identical. A configuration value such as epsilon = 0.1 is therefore incomplete unless the implementation’s convention is known.
The gradient expression exposes the central effect. Once the model assigns the target class a probability above its smoothed target value, the sign of that target-class logit gradient changes. Cross-entropy no longer keeps rewarding movement toward probability one in the same manner as a one-hot target.
Smoothing changes the optimum, not only the loss scale
Label smoothing is sometimes described as if it merely weakens the training signal. That misses a structural detail: the target distribution itself has changed.
Multiplying ordinary cross-entropy by a constant scales its gradients but leaves the loss minimizer unchanged. Replacing one-hot targets with smoothed targets changes the probabilities that minimize cross-entropy for each target distribution. The model is being asked to fit a different target.
Consider a four-class example using the convention that spreads epsilon across all classes. With epsilon = 0.2, the target becomes:
one-hot: [1.00, 0.00, 0.00, 0.00]
smoothed: [0.85, 0.05, 0.05, 0.05]If the model predicts:
p = [0.90, 0.04, 0.03, 0.03]then the target-class component of p - q is positive under the smoothed target:
0.90 - 0.85 = 0.05That gradient pushes the target logit downward under gradient descent, rather than pushing it further upward. The non-target entries also receive gradients relative to their nonzero target mass.
This behavior is distinct from changing optimizer settings, weight decay, or a global loss multiplier. Those choices can alter parameter updates, but they do not define the same target distribution.
Uniform smoothing encodes a specific assumption
Uniform label smoothing assigns the reserved mass according to a fixed rule that does not depend on semantic similarity between classes. In a ten-class problem, every non-target class can receive the same target mass even if some class pairs are much more closely related than others.
That makes uniform smoothing deliberately simple. It does not express a claim that a particular alternative class is plausible for a particular example. The extra mass is a training target design choice.
This distinction matters when labels have known structure. A hand-built soft target that assigns more mass to related classes is not merely label smoothing with a different coefficient. It carries additional information about class relationships. Likewise, targets produced by another model introduce that model’s distribution rather than a uniform prior over alternatives.
Developers should keep these mechanisms separate in experiment metadata. Calling every non-one-hot target “smoothing” can hide a material change in what information reaches the loss.
Noisy labels and uncertain labels are different cases
A smoothed target can reduce the magnitude of the push toward a recorded class, but it does not identify an incorrect annotation. Every example receives the same transformation when a global smoothing coefficient is used.
Suppose an example is assigned class A even though its correct class is B. Uniform smoothing still gives the largest target mass to A. The loss signal is less extreme than a one-hot target, yet its main direction still favors the recorded class.
Uncertainty in the task also needs separate treatment. If an input genuinely admits multiple valid labels, a probability distribution derived from the annotation process can represent that ambiguity directly. Uniform smoothing cannot infer those example-specific proportions.
Label smoothing can therefore alter sensitivity to hard targets without serving as a general correction for annotation errors or ambiguous ground truth.
Confidence values need separate evaluation
Because label smoothing discourages extreme training targets, it can change the distribution of predicted probabilities. That does not make it a calibration guarantee.
Calibration concerns the relationship between predicted confidence and observed outcomes under a defined evaluation setup. Label smoothing modifies training. Temperature scaling, by contrast, is a post-training transformation fitted on held-out data that rescales logits without changing their ordering when a single positive temperature is used.
The two mechanisms act at different stages and solve different optimization problems. A model trained with smoothed labels can still require calibration assessment for an application that uses probability thresholds. Conversely, a model trained with one-hot labels can be calibrated after training without changing the original target distribution.
For systems that route requests, reject low-confidence predictions, or attach probabilities to user-visible outputs, classification accuracy alone does not establish that a chosen confidence threshold has the intended meaning.
Smoothing interacts with other soft-target objectives
Label smoothing becomes easy to misapply when the loss already contains a soft target. Knowledge distillation is a common example: a student may be trained against a teacher distribution, sometimes alongside a hard-label term.
Adding uniform smoothing to the hard-label component changes that component’s target. Applying smoothing to the teacher distribution would be a different operation again. These choices should not be treated as interchangeable because they alter separate terms in the objective.
Mixup-style training creates another distinction. When two examples and their labels are mixed, the resulting target can already contain mass on multiple classes according to the mixing coefficient. Adding uniform smoothing afterward further redistributes that target.
The implementation question is therefore not simply whether smoothing is enabled. It is which target enters each loss term after all target transformations have been applied.
Class count affects the redistributed mass
The same epsilon does not imply the same per-class non-target mass across tasks with different numbers of classes.
Using the all-class convention:
non_target_mass_per_class = epsilon / KWith epsilon = 0.1, a five-class classifier assigns 0.02 to each non-target entry through the uniform component, while a thousand-class classifier assigns 0.0001. The total smoothing mass is controlled by epsilon, but its distribution across individual alternatives depends on K.
This matters when comparing configurations across datasets or heads. Copying a coefficient preserves the aggregate smoothing parameter under the same convention, not the exact target assigned to each non-target class.
It also makes implementation tests useful. Given a known class count, target index, and coefficient, a small unit test can verify the exact target distribution or loss value expected from the framework. That catches convention mismatches without relying on training curves to reveal them later.
The coefficient belongs to the model specification
Label smoothing changes the objective that shapes the classifier’s logits. It should be recorded with the same care as the loss definition, class mapping, and output activation assumptions.
A checkpoint alone may not reveal whether its output geometry was produced from one-hot targets, uniformly smoothed targets, or another soft-target scheme. That distinction can matter when reproducing training, comparing confidence behavior, or combining the model with later calibration.
The useful boundary is precise: label smoothing is a target-distribution modification inside classification training. It can reduce the objective’s pressure toward extreme probabilities, but it does not repair incorrect labels, encode semantic class relations, or establish calibrated confidence on its own.