A classification model is often trained as if the correct class deserves all of the target probability and every other class deserves none. For a three-class problem, an example labeled cat might therefore use this target:

cat: 1.00
dog: 0.00
fox: 0.00

That target is convenient, but it asks the model to push probability toward an extreme even when labels are imperfect, classes overlap, or the input is genuinely ambiguous. Label smoothing changes the training target so that a small amount of probability mass is assigned away from the labeled class.

This article develops the idea from the loss function upward. You will see exactly what label smoothing changes, why that changes the gradient seen during training, and why the technique should be evaluated rather than treated as a general cure for overconfidence.

Start with the target distribution

Consider a classifier with three classes:

cat
dog
fox

For a training image labeled cat, ordinary one-hot encoding gives the target distribution:

[1.0, 0.0, 0.0]

A simple form of label smoothing with smoothing parameter ε = 0.1 instead uses:

[0.9333, 0.0333, 0.0333]

One common definition is:

y_smooth = (1 - ε) * y_one_hot + ε / K

where K is the number of classes. With K = 3, the labeled class receives:

1 - 0.1 + 0.1 / 3 = 0.9333...

and each class, including the labeled class through the uniform component, receives 0.1 / 3 from the smoothing distribution.

You may also encounter implementations that assign 1 - ε to the labeled class and distribute ε only across the other K - 1 classes. That convention produces different numerical targets. When comparing libraries or experiments, check the exact definition instead of assuming that the same ε means the same target distribution.

The important mental model is simple: the training target is no longer a point mass on one class.

How smoothing changes cross-entropy training

For a target distribution y and model probabilities p, multiclass cross-entropy is:

L = -sum(y_i * log(p_i))

With a one-hot target, only the labeled class contributes directly to this sum because all other target values are zero. If cat is the label, the loss reduces to:

L = -log(p_cat)

Increasing p_cat therefore keeps reducing this loss as it approaches 1.

With label smoothing, the other classes have small non-zero target probabilities. Pushing p_cat extremely close to 1 necessarily pushes the remaining probabilities toward zero, which becomes costly because those classes now appear in the cross-entropy target.

For softmax logits z, the cross-entropy gradient has a particularly useful form:

dL / dz_i = p_i - y_i

Suppose a three-class model predicts:

[0.98, 0.01, 0.01]

for a cat example. With the ordinary target [1, 0, 0], the logit gradients are:

[-0.02, 0.01, 0.01]

With the smoothed target [0.9333, 0.0333, 0.0333], they are approximately:

[ 0.0467, -0.0233, -0.0233]

The sign has changed for the labeled class: at this point, the smoothed objective considers the prediction more concentrated than its target. Training therefore does not keep rewarding arbitrarily extreme separation on that example.

This is the mechanism to remember. Label smoothing does not directly edit predictions after training. It changes the objective that generates training gradients.

Why the technique can help

Neural classifiers can fit training labels with very large differences between logits. Once an example is already classified correctly, ordinary one-hot cross-entropy can still reward increasing the labeled class probability toward 1.

Label smoothing weakens that incentive. Depending on the model, data, and optimization setup, this can act as a useful regularizing influence and may improve generalization.

It can also make the model less prone to extremely concentrated output distributions. That property is sometimes described loosely as making a model “less confident,” but the phrase needs care. A lower maximum softmax probability is not automatically a better probability estimate, and label smoothing is not a guarantee of calibrated confidence.

The practical question is not whether smoothing makes numbers smaller. It is whether the resulting model performs better on the properties your application actually needs.

Label smoothing is not label-noise repair

A common mistake is to treat smoothing as a way to fix incorrect labels.

Suppose an image is actually a fox but is mislabeled as a cat. With one-hot training, the target is:

cat: 1.00
dog: 0.00
fox: 0.00

With smoothing, it might become:

cat: 0.9333
dog: 0.0333
fox: 0.0333

The wrong class still receives by far the largest target probability. Smoothing reduces the extremity of the training signal, but it does not discover that the annotation is wrong.

If incorrect labels are a major source of error, investigate annotation quality, ambiguous examples, class definitions, and disagreement between annotators. Label smoothing may reduce sensitivity to some training examples, but it is not a substitute for fixing a broken labeling process.

Do not confuse smoothing with naturally soft labels

Sometimes the uncertainty in a target is meaningful information rather than a regularization choice.

Imagine five qualified annotators classify an image and produce these votes:

cat: 4
dog: 1
fox: 0

A target such as:

[0.8, 0.2, 0.0]

contains information about actual disagreement. Uniform label smoothing would instead move probability toward every class according to a fixed rule, regardless of which alternative annotators considered plausible.

These ideas serve different purposes:

  • Label smoothing deliberately modifies hard targets according to a chosen smoothing rule.
  • Soft labels can encode observed uncertainty, annotator disagreement, or another teacher signal.

When trustworthy soft targets are available, replacing their structure with uniform smoothing can throw useful information away.

Choose the smoothing strength as a hyperparameter

The smoothing parameter ε controls how far the target moves away from one-hot encoding.

At ε = 0, there is no smoothing. As ε increases under the uniform-mixture definition, the target moves closer to a uniform distribution. Excessive smoothing makes the training signal less specific because the labeled class becomes less distinct from alternatives.

That creates a real trade-off. A small amount of smoothing may regularize a model that otherwise becomes too concentrated on its training labels. Too much can weaken class separation and hurt the metric that matters to the application.

There is therefore no universal smoothing value that should be copied between tasks. Treat ε like other training hyperparameters: compare candidate values on validation data that is separate from the data used to fit model parameters.

For imbalanced or cost-sensitive problems, also inspect whether uniform smoothing matches the semantics you want. Uniformly allocating probability to every class is a mathematical training choice; it does not mean all classification mistakes have equal real-world meaning.

Evaluate more than top-1 accuracy

If label smoothing is introduced because confidence values matter, evaluating only classification accuracy misses part of the reason for the change.

A useful experiment keeps the rest of the training setup as stable as practical and compares an unsmoothed baseline with one or more smoothing strengths. Depending on the application, evaluate:

  • validation loss and task accuracy or another primary quality metric;
  • precision and recall when class-specific errors matter;
  • the distribution of predicted probabilities;
  • calibration with reliability diagrams or an appropriate calibration metric;
  • performance on important classes or subgroups;
  • downstream decisions made from probability thresholds.

These measurements answer different questions. A model can improve accuracy while becoming worse calibrated, or improve a calibration summary while hurting recall on an important class.

If calibrated probabilities are required for decisions, evaluate calibration directly on held-out data. Post-hoc calibration methods solve a different problem: they learn a mapping from model scores to probability estimates after the classifier has been trained. Label smoothing instead changes the training objective itself.

Watch for interactions with distillation and other objectives

Real training pipelines often combine several losses. For example, knowledge distillation may train a student model using both hard labels and a teacher’s probability distribution.

If you also smooth the hard-label component, keep the roles separate in your reasoning:

total loss
  = weight_a * hard-label loss
  + weight_b * teacher loss

Smoothing changes the target inside the hard-label term. The teacher distribution carries its own information about relationships among classes. Applying transformations without tracking which target they affect can make experiments difficult to interpret.

The same principle applies to class weighting, focal-style losses, or custom cost-sensitive objectives. Label smoothing is not an isolated switch once the loss function has multiple components. Write down the complete objective and verify how each transformation changes it.

Common mistakes

Assuming lower confidence means better calibration

A model that predicts 0.7 instead of 0.99 is not necessarily more calibrated. Calibration depends on whether predicted probabilities match observed frequencies. Measure it rather than inferring it from less extreme outputs.

Comparing smoothing parameters across different definitions

ε = 0.1 can produce different targets depending on whether the smoothing mass includes the labeled class. Verify the library’s formula before reproducing an experiment or porting a configuration.

Using smoothing to hide bad labels

Smoothing may soften the effect of labels, but systematic annotation errors still teach the wrong relationship. Fix data problems when you can identify them.

Smoothing already meaningful targets without a reason

If targets represent real probability distributions or annotator disagreement, uniform smoothing changes that information. Make sure the transformation matches the goal.

Tuning on the test set

Choose smoothing strength with training and validation data. Keep the test set for the final estimate of generalization so repeated tuning does not leak information from it into model choices.

When label smoothing is worth trying

Label smoothing is a reasonable experiment when you train a multiclass classifier with hard labels and one-hot cross-entropy, especially when the model fits the training data with very concentrated predictions and you want to test whether a less extreme objective improves held-out behavior.

It is less compelling when the baseline already generalizes well, when you possess meaningful soft targets that should be preserved, or when the main problem is systematic label corruption. It is also unnecessary when confidence values are irrelevant and validation experiments show no benefit to the primary task metric.

Keep an unsmoothed baseline. The value of label smoothing is empirical: it changes the inductive bias of training, and whether that change helps depends on the task.

Conclusion

Label smoothing replaces an exact one-hot training target with a distribution that reserves some probability mass away from the labeled class. In cross-entropy training, that changes the gradients and reduces the incentive to drive the labeled class probability toward an extreme value on every example.

The technique is simple, but its interpretation matters. It does not repair incorrect labels, does not guarantee calibrated probabilities, and should not erase meaningful soft-target information. Use it as a controlled training choice: verify the implementation’s definition, tune the smoothing strength on validation data, and measure the outcomes your application actually depends on.

Artificial Intelligence Machine Learning Classification Model Training