A classifier trained with ordinary cross-entropy is usually given a hard target: the correct class has probability 1, and every other class has probability 0.

For a three-class example:

cat    dog    bird
1.0    0.0    0.0

That target is simple and often appropriate. But it also asks the model to keep increasing the correct-class logit relative to the others even after the prediction is already very confident.

Label smoothing changes the training target so a small amount of probability mass is spread across classes instead of placing all of it on the labeled class.

The technique can act as a regularizer for classification models, but it is not automatically beneficial. It changes the objective itself, so it can also weaken useful confidence information, interact with noisy labels, and be a poor fit when exact target probabilities carry meaning.

The practical goal is therefore not “always enable label smoothing.” It is to understand exactly what loss you are asking the model to minimize.

Start with the target distribution

Suppose a classifier has three classes and the correct class is cat.

With ordinary one-hot training, the target distribution is:

cat    dog    bird
1.0    0.0    0.0

With label smoothing, the target becomes a mixture of:

  • the original hard target;
  • a uniform distribution over all classes.

If the smoothing amount is epsilon, the mixture is:

smoothed_target
    = (1 - epsilon) * one_hot_target
    + epsilon * uniform_distribution

For three classes and epsilon = 0.2:

uniform distribution = [1/3, 1/3, 1/3]

smoothed target
= 0.8 * [1, 0, 0] + 0.2 * [1/3, 1/3, 1/3]
= [0.8667, 0.0667, 0.0667]

The class label has not changed. cat is still the target class. What changed is how strongly the loss rewards pushing its predicted probability toward exactly 1.

This is the central mental model for label smoothing.

Cross-entropy works with probability targets

For predicted class probabilities p and target probabilities q, cross-entropy is:

loss = -sum(q_i * log(p_i))

With a one-hot target, only the correct class contributes:

q = [1, 0, 0]

loss = -log(p_cat)

With a smoothed target, every class contributes:

q = [0.8667, 0.0667, 0.0667]

loss =
    -0.8667 * log(p_cat)
    -0.0667 * log(p_dog)
    -0.0667 * log(p_bird)

This matters because extremely small probabilities for the non-target classes now increase the loss.

Ordinary cross-entropy says, in effect:

make the labeled class probability as large as possible

Label-smoothed cross-entropy says:

prefer the labeled class strongly,
but do not make the full target distribution infinitely sharp

The second statement is deliberately softer.

The smallest useful PyTorch example

PyTorch supports label smoothing directly in CrossEntropyLoss:

import torch
from torch import nn

logits = torch.tensor([
    [2.0, 0.0, -1.0],
])

target = torch.tensor([0])

loss_fn = nn.CrossEntropyLoss(label_smoothing=0.2)
loss = loss_fn(logits, target)

print(loss.item())

The input is still logits, not probabilities. CrossEntropyLoss applies the appropriate log-softmax behavior internally.

The target is still the integer class index 0. PyTorch constructs the smoothed target as part of the loss calculation.

For three classes with smoothing 0.2, the effective target is:

[0.8667, 0.0667, 0.0667]

PyTorch defines its label_smoothing behavior as mixing the original ground-truth target with a uniform distribution over all classes.

Why the exact smoothing formula matters

Not every description of label smoothing uses the same convention.

A common alternative formula assigns:

correct class     = 1 - epsilon
incorrect classes = epsilon / (C - 1)

For three classes and epsilon = 0.2, that would produce:

[0.8, 0.1, 0.1]

PyTorch’s built-in label_smoothing=0.2 instead mixes with a uniform distribution across all C classes:

correct class     = (1 - epsilon) + epsilon / C
incorrect classes = epsilon / C

For three classes:

[0.8667, 0.0667, 0.0667]

Both are forms of smoothing, but they are not numerically identical.

When reproducing a paper, porting code between frameworks, or comparing experiments, verify the exact convention instead of assuming that the same epsilon means the same target distribution everywhere.

Check the built-in loss against the formula

The built-in behavior can be reproduced manually:

import torch
import torch.nn.functional as F

logits = torch.tensor([
    [2.0, 0.0, -1.0],
])

target = torch.tensor([0])
epsilon = 0.2

built_in = F.cross_entropy(
    logits,
    target,
    label_smoothing=epsilon,
)

class_count = logits.shape[1]

smoothed = torch.full_like(
    logits,
    epsilon / class_count,
)

smoothed[0, target.item()] += 1 - epsilon

log_probabilities = F.log_softmax(logits, dim=1)

manual = -(smoothed * log_probabilities).sum(dim=1).mean()

print(built_in.item())
print(manual.item())

The two values should agree apart from normal floating-point rounding.

This example is useful for more than testing PyTorch. It makes the objective visible: label smoothing changes the target distribution used by the loss.

What happens to the gradient

For softmax followed by cross-entropy with a target distribution q, the gradient with respect to a logit is:

gradient_i = p_i - q_i

where p_i is the model’s predicted probability for class i.

For one-hot training, the target for the correct class is 1. If the model predicts 0.99, the corresponding gradient term is:

0.99 - 1.00 = -0.01

The objective still pushes the correct-class logit upward.

With a smoothed correct-class target of 0.8667:

0.99 - 0.8667 = 0.1233

Now the sign has changed. For that training example, the loss would push against making that class even more dominant.

This is the mechanism behind label smoothing’s effect on extreme confidence. It is not a post-processing adjustment to predictions. It directly changes training gradients.

Label smoothing is not the same as changing predictions at inference time

Label smoothing belongs to the training objective.

A model trained with smoothing still produces ordinary logits at inference time:

with torch.no_grad():
    logits = model(inputs)
    probabilities = torch.softmax(logits, dim=1)

You do not normally add epsilon to output probabilities at inference time.

The smoothing was already reflected in the learned parameters because it changed the loss during training.

This distinction is important when evaluating or deploying a model. Training-time label smoothing and inference-time probability calibration are separate operations.

Use validation metrics that match the real task

Because label smoothing changes the objective, training loss is not directly comparable to unsmoothed training loss in the naive sense.

Imagine two models with equally correct class rankings. The model trained without smoothing can achieve a lower hard-label cross-entropy by becoming extremely confident. The smoothed model is intentionally discouraged from optimizing toward those same extreme targets.

For model selection, use metrics that represent the actual requirement:

classification quality -> accuracy, F1, precision, recall
ranking quality        -> task-specific ranking metrics
probability quality    -> log loss, Brier score, calibration analysis

Do not conclude that a smoothed model is worse merely because its training-loss curve has a different scale or optimum.

Also evaluate on unsmoothed ground-truth labels unless your real evaluation target is genuinely a soft distribution.

Label smoothing can help with overconfident classifiers

Neural classifiers can produce highly concentrated softmax outputs. A correct prediction such as:

[0.9999, 0.00005, 0.00005]

and a less extreme prediction such as:

[0.90, 0.06, 0.04]

have the same top-1 class.

Yet ordinary one-hot cross-entropy continues to reward pushing the first probability closer to 1.

Label smoothing reduces that incentive.

This can be useful when extreme confidence is not required by the application and when reducing overfitting improves held-out performance.

But “less confident” does not automatically mean “well calibrated.” Calibration means that predicted probabilities correspond appropriately to observed frequencies. Label smoothing can influence calibration, but it is not a universal calibration method.

If probability quality matters operationally, measure it directly.

Do not use smoothing as a substitute for fixing bad labels

Suppose training data contains mislabeled examples.

Label smoothing may reduce the gradient pressure created by any single hard target, including a wrong one. That can make training less sensitive to some label errors.

But the incorrect example still points toward the wrong class.

A record labeled cat when the image is actually a dog might become:

cat    dog    bird
0.87   0.07   0.07

That is softer than [1, 0, 0], but it still says cat should receive most of the probability.

If the dataset has systematic annotation problems, duplicated classes, broken label mapping, or distribution shift, fix those issues directly. Label smoothing is not a data-cleaning algorithm.

Be careful when labels are already probabilistic

Some tasks naturally have soft targets.

Examples include:

  • labels aggregated from multiple annotators;
  • distillation targets from another model;
  • probability distributions representing genuine ambiguity;
  • blended targets created by techniques such as Mixup.

If the target already contains meaningful probabilities, applying additional generic smoothing changes those probabilities again.

That may be useful, but it should be intentional.

Ask:

Does the target distribution already encode uncertainty I care about?

If yes, blindly mixing it toward uniform can erase some of that information.

In PyTorch, CrossEntropyLoss can also accept class-probability targets. When using probability targets, verify that they represent valid distributions; PyTorch documents that it does not strictly validate all probability constraints for you.

Class imbalance and label smoothing solve different problems

Label smoothing spreads some probability mass across classes. It does not rebalance how often classes appear in the dataset.

Suppose 95% of samples belong to class A and 5% belong to class B. Smoothing each target does not make the training set balanced.

Class imbalance may require techniques such as:

  • collecting better-balanced data;
  • resampling;
  • class weighting;
  • using a metric appropriate for the minority class.

PyTorch’s CrossEntropyLoss supports both class weights and label smoothing, but they address different concerns.

Do not use a smoothing value as if it were an imbalance-control parameter.

Smoothing can hurt when fine distinctions matter

A large smoothing value deliberately weakens the preference for the labeled class.

That can be harmful when the task requires the model to learn very sharp distinctions and the labels are already reliable.

For example, with C = 10 and epsilon = 0.5, the PyTorch target for the correct class is:

(1 - 0.5) + 0.5 / 10 = 0.55

Each incorrect class receives:

0.5 / 10 = 0.05

The correct class still has the largest target probability, but the supervision is much weaker than a one-hot target.

Increasing epsilon is therefore not a monotonic path to better regularization. Too much smoothing can produce under-confident or underfit behavior and can reduce accuracy.

Treat the smoothing amount as a hyperparameter, not a safety switch.

Start with a small controlled experiment

A useful experiment changes only the smoothing setting while keeping the rest of training fixed.

For example:

from torch import nn

baseline_loss = nn.CrossEntropyLoss(
    label_smoothing=0.0,
)

smoothed_loss = nn.CrossEntropyLoss(
    label_smoothing=0.1,
)

Train otherwise comparable runs and inspect:

validation accuracy
validation loss
per-class metrics
confidence distribution
calibration metrics, if probabilities matter

The exact useful value depends on the model, dataset, label quality, and training recipe.

A result from one image-classification benchmark is not a guarantee that the same smoothing amount is appropriate for text, audio, medical, ranking, or small-data classification.

Watch the interaction with other regularization

Training pipelines often already contain several regularizers:

weight decay
data augmentation
dropout
Mixup or CutMix
early stopping
label smoothing

These techniques do not all act in the same way, but their effects can overlap.

For example, strong augmentation may already make the classification problem substantially harder. Mixup already creates soft targets. Adding aggressive label smoothing on top may reduce supervision more than intended.

When several regularizers are changed at once, it becomes difficult to know which one helped.

Introduce smoothing as a controlled change and evaluate the combination rather than assuming each regularizer contributes an independent improvement.

Common mistakes

Applying softmax before CrossEntropyLoss

This is unnecessary and changes the numerical path.

Prefer:

loss = loss_fn(logits, target)

rather than:

probabilities = torch.softmax(logits, dim=1)
loss = loss_fn(probabilities, target)

CrossEntropyLoss expects unnormalized logits.

Assuming epsilon is distributed only among wrong classes

That convention exists, but PyTorch’s built-in option mixes the target with a uniform distribution over all classes.

Check the framework or implementation you are actually using.

Comparing smoothed and unsmoothed training loss as identical objectives

They are different losses because their target distributions differ.

Compare task-level validation results, not only the raw training-loss numbers.

Using smoothing to repair systematic annotation errors

Smoothing weakens targets; it does not identify or correct wrong labels.

Choosing a large value because a small value helped

More smoothing means less concentrated supervision. Past some point it can reduce useful learning.

When label smoothing is a good candidate

Consider label smoothing when:

  • you are training a standard multi-class classifier;
  • the model becomes extremely confident on training examples;
  • validation performance suggests overfitting;
  • labels are mostly reliable but absolute one-hot certainty is not essential;
  • you can compare controlled validation runs.

It is less compelling when:

  • targets already represent meaningful probability distributions;
  • the task depends on preserving exact probability targets;
  • the model is already underfitting;
  • the dataset has structural label errors that need correction;
  • strong soft-target regularization is already part of the training recipe.

Keep the objective visible

Label smoothing is easiest to reason about when you treat it as a change to the target distribution rather than as a vague anti-overconfidence trick.

For PyTorch, label_smoothing=epsilon mixes a one-hot target with a uniform distribution over all classes. That changes the cross-entropy terms and therefore changes the gradients used during training.

Start with a modest controlled experiment, keep inference unchanged, and evaluate the metrics that matter for the application. If smoothing improves held-out behavior, keep it because the evidence supports it—not because softer labels are inherently better.

Artificial Intelligence Machine Learning Neural Networks Training