Understand Cross-Entropy Loss for Classification

A classifier needs more than a way to count correct answers. During training, it needs a signal that says not only whether a prediction was wrong, but also how the model’s scores should change.

Suppose the correct class is cat. A model that assigns cat probability 0.49 and another class 0.51 is wrong, but it is close to the decision boundary. A model that assigns cat probability 0.001 is also wrong, and much more confident in that mistake. Treating those predictions as equally bad throws away useful information.

Cross-entropy loss solves this problem by scoring the probability assigned to the correct class. It gives a small loss when that probability is high and a rapidly increasing loss when it approaches zero. Combined with softmax, it is a standard objective for training single-label multiclass classifiers.

This article builds cross-entropy from one prediction, connects it to logits and softmax, and explains the practical details that commonly cause incorrect implementations or misleading interpretations.

Start with the probability of the correct class

Consider a classifier with three possible labels:

cat
dog
rabbit

For one image, suppose the correct label is cat and the model predicts:

P(cat)    = 0.70
P(dog)    = 0.20
P(rabbit) = 0.10

The probabilities sum to 1. For ordinary single-label classification, cross-entropy only needs the probability assigned to the correct class for this example.

If y is the correct class and p_y is its predicted probability, the loss is:

loss = -log(p_y)

Using the natural logarithm:

loss = -log(0.70)
     ≈ 0.357

Now compare several probabilities for the correct class:

p_y = 0.90 -> loss ≈ 0.105
p_y = 0.70 -> loss ≈ 0.357
p_y = 0.50 -> loss ≈ 0.693
p_y = 0.10 -> loss ≈ 2.303
p_y = 0.01 -> loss ≈ 4.605

Two properties are immediately useful.

First, increasing the correct class probability lowers the loss. The minimum is approached as p_y approaches 1.

Second, the penalty is nonlinear. Moving the correct class from 0.10 to 0.01 increases the loss much more than a simple wrong-answer count would reveal. A confident mistake therefore creates a strong training signal.

That is the core mental model: cross-entropy asks how much probability the model gave to what actually happened.

Why the logarithm is useful

The negative logarithm can look arbitrary until you consider how probabilities combine.

For independent observations, joint probabilities multiply. If a model assigns probabilities p1, p2, and p3 to the correct labels of three examples, their likelihood is proportional to:

p1 * p2 * p3

Taking logarithms turns the product into a sum:

log(p1 * p2 * p3)
= log(p1) + log(p2) + log(p3)

Maximizing the log-likelihood is therefore equivalent to minimizing its negative:

-[log(p1) + log(p2) + log(p3)]

Averaging those negative log-probabilities gives the usual mean cross-entropy over a batch.

This connection matters because cross-entropy is not merely a convenient penalty curve. Under the usual classification setup, minimizing it corresponds to maximizing the likelihood of the observed labels under the model.

From logits to probabilities

Neural classifiers normally do not produce probabilities directly. Their final layer produces unrestricted real-valued scores called logits.

For example:

cat:     2.0
dog:     1.0
rabbit: -1.0

A logit can be positive or negative and the logits do not need to sum to anything. To interpret them as a categorical probability distribution, apply softmax:

softmax(z_i) = exp(z_i) / sum_j exp(z_j)

For the logits above, the exponentials are approximately:

exp(2.0)  ≈ 7.389
exp(1.0)  ≈ 2.718
exp(-1.0) ≈ 0.368

Their sum is about 10.475, so the probabilities are approximately:

P(cat)    ≈ 0.705
P(dog)    ≈ 0.259
P(rabbit) ≈ 0.035

If cat is the target, the cross-entropy is approximately:

-log(0.705) ≈ 0.350

Conceptually, the pipeline is:

input
  -> model
  -> logits
  -> softmax probabilities
  -> probability of target class
  -> negative log
  -> loss

In actual training code, however, you usually should not implement those last steps separately.

Compute cross-entropy from logits, not rounded probabilities

A numerically robust loss implementation accepts logits and combines the softmax and logarithm internally. This avoids an important numerical problem.

A direct implementation might appear to do this:

probabilities = softmax(logits)
loss = -log(probabilities[target])

Mathematically, that is correct. Numerically, very large differences between logits can cause exponentials to overflow or tiny probabilities to underflow toward zero.

Stable implementations use the log-sum-exp identity. For target class y, cross-entropy can be written directly from logits as:

loss = -z_y + log(sum_j exp(z_j))

Subtracting the largest logit before exponentiation leaves the result unchanged while keeping intermediate values in a safer numerical range:

m = max(z)
loss = -z_y + m + log(sum_j exp(z_j - m))

This is why common machine-learning loss functions generally expect raw logits. Applying softmax yourself before a loss that already performs the corresponding normalization can produce the wrong objective.

The exact API depends on the framework, so check whether a particular loss expects logits, log-probabilities, or probabilities rather than assuming from its name.

Cross-entropy uses the whole distribution during optimization

It is easy to hear that cross-entropy “only looks at the correct class” and conclude that the other logits do not matter. That is not true once softmax is included.

The loss for target class y is:

loss = -log(softmax(z)_y)

The target probability depends on every logit because all exponentials appear in the softmax denominator. Increasing a competing class logit can lower the target probability even when the target logit stays unchanged.

The gradient makes this especially clear. For class k, the derivative of softmax cross-entropy with respect to logit z_k is:

dL/dz_k = p_k - 1[k = y]

where 1[k = y] is 1 for the correct class and 0 otherwise.

For the target class:

dL/dz_y = p_y - 1

If the target probability is too low, this value is negative. Gradient descent therefore pushes the target logit upward.

For every non-target class:

dL/dz_k = p_k

Those gradients are positive, so gradient descent pushes competing logits downward. A wrong class with a large probability receives a larger correction than a wrong class that already has little probability mass.

This simple gradient is one reason softmax cross-entropy works well as a training objective: the update direction follows the model’s current distribution rather than only its final argmax decision.

Extend the idea to a batch

Training normally evaluates many examples at once. Suppose a batch contains three target probabilities:

example 1: p_y = 0.80
example 2: p_y = 0.60
example 3: p_y = 0.10

Their individual losses are approximately:

0.223
0.511
2.303

The mean loss is:

(0.223 + 0.511 + 2.303) / 3
≈ 1.012

Notice how the confident mistake in the third example contributes most of the batch loss. This is useful when the example is genuinely informative, but it also explains why mislabeled or corrupted examples can have disproportionate influence: a model that strongly disagrees with an incorrect target can receive a very large loss.

Reduction also matters. A framework may return the mean, the sum, or individual per-example losses. Changing from a mean to a sum changes the gradient scale with batch size unless the rest of the optimization setup compensates for it.

When comparing training runs, confirm that the reduction convention is the same.

One-hot notation describes the same loss

Cross-entropy is often written with a target distribution instead of a target class index.

For three classes, the target cat can be represented as a one-hot vector:

y = [1, 0, 0]

If the model probabilities are:

p = [0.70, 0.20, 0.10]

categorical cross-entropy is:

L = -sum_i y_i log(p_i)

Substituting the values:

L = -(1 * log(0.70)
    + 0 * log(0.20)
    + 0 * log(0.10))

which reduces to:

L = -log(0.70)

The one-hot form is useful because it generalizes naturally to soft targets, where more than one class can receive nonzero target probability. Techniques such as label smoothing and knowledge distillation can use such target distributions, although their exact objectives and implementation details vary.

For ordinary hard-label multiclass classification, storing an integer class index is usually simpler than materializing one-hot vectors.

Do not confuse multiclass and multilabel classification

A major implementation mistake is using the wrong form of cross-entropy for the prediction problem.

In single-label multiclass classification, exactly one class is the target. Examples include choosing one animal species or one document category. Softmax is appropriate because the classes compete for a probability mass that sums to 1.

In multilabel classification, several labels can be true at the same time. A photo might contain both person and bicycle. Those outputs should not be forced to compete through one softmax distribution.

A common multilabel setup treats each label as a separate binary prediction and uses a sigmoid-based binary cross-entropy objective for each output.

The distinction is structural:

multiclass:
exactly one class is true
-> softmax across classes
-> categorical cross-entropy

multilabel:
zero, one, or several labels may be true
-> independent sigmoid outputs
-> binary cross-entropy per label

Using softmax for a genuinely multilabel problem forces increasing one label’s probability to decrease others, even when several labels should be simultaneously likely.

Accuracy and cross-entropy measure different things

Two models can have identical accuracy and very different cross-entropy.

Suppose both correctly classify an example as cat:

model A: P(cat) = 0.51
model B: P(cat) = 0.95

Both receive one correct classification if cat is the largest probability. Their losses differ substantially:

model A: -log(0.51) ≈ 0.673
model B: -log(0.95) ≈ 0.051

Now consider a wrong prediction:

P(cat) = 0.01
P(dog) = 0.98
P(rabbit) = 0.01

If cat is correct, the loss is about 4.605. Accuracy records only another error; cross-entropy records that the model was extremely confident in the wrong class.

This does not make cross-entropy a universally better evaluation metric. It answers a different question. Accuracy is useful when the final class decision is what matters. Cross-entropy is sensitive to the full predictive probabilities and is useful when probability quality matters or when monitoring the objective used for training.

A lower cross-entropy also does not guarantee that probabilities are well calibrated for a particular deployment decision. Calibration should be evaluated directly when reliable confidence estimates matter.

Know the baseline before interpreting a loss value

A raw cross-entropy value has little meaning without context.

For a balanced K-class problem, a uniform predictor assigns probability 1/K to every class. Its cross-entropy per example is:

-log(1 / K) = log(K)

For four classes:

log(4) ≈ 1.386

For one hundred classes:

log(100) ≈ 4.605

So a loss of 1.0 has very different implications in a four-class task and a hundred-class task.

Even log(K) is only a simple reference point, not a universal benchmark. Class imbalance changes useful baselines, label noise can impose an irreducible error floor, and different weighting or smoothing schemes change the objective being reported.

Compare loss values only when the task, target construction, weighting, reduction, and data distribution are sufficiently comparable.

Class weighting changes what the loss means

When some mistakes matter more than others, training may assign different weights to classes or examples.

A simplified weighted loss for target class y is:

weighted_loss = w_y * -log(p_y)

If rare class A receives weight 4 and common class B receives weight 1, an equally confident mistake on A contributes four times as much to the unreduced objective.

That can help redirect optimization toward underrepresented or high-cost cases, but it changes the training objective. The resulting loss is no longer directly comparable with an unweighted cross-entropy value.

Weights also do not automatically solve every class-imbalance problem. They can increase gradient variance or trade performance between groups. Choose them according to the decision goal and evaluate per-class metrics rather than assuming a lower weighted loss implies better behavior everywhere.

Common mistakes to avoid

Applying softmax twice

If a loss function expects logits and performs log-softmax internally, pass logits. Feeding already-softmaxed probabilities changes the values being normalized and therefore changes the objective.

Taking the logarithm of a rounded probability

Do not round probabilities before computing a loss. A small value rounded to zero would imply -log(0), which is unbounded. Stable logit-based implementations avoid this path.

Using categorical cross-entropy for independent labels

If multiple labels may be true simultaneously, a single softmax distribution is usually the wrong model of the target. Use an objective that matches independent binary targets or another formulation designed for the task.

Comparing losses with different reductions or weights

A summed loss, a mean loss, and a class-weighted mean can have very different scales. Record the exact objective when comparing experiments.

Treating low training loss as proof of generalization

A sufficiently flexible model can reduce training cross-entropy while becoming worse on unseen data. Monitor a held-out validation set and task-relevant metrics. Cross-entropy is an optimization objective, not evidence by itself that the deployed model will perform well.

When cross-entropy is the right tool

Softmax cross-entropy is a natural choice when each example belongs to exactly one of several mutually exclusive classes and the model produces one logit per class. It gives a smooth training signal, penalizes confident errors strongly, and has a direct likelihood interpretation.

A different objective may be more appropriate when the task structure differs. Multilabel problems need a formulation that allows multiple simultaneous positives. Severe imbalance may justify weighting or a specialized loss after careful evaluation. Regression requires an objective for continuous targets. Ranking and metric-learning problems may be better represented by pairwise, listwise, or contrastive objectives.

The important principle is not to choose cross-entropy because classification code commonly uses it. Choose it because its probabilistic assumptions match the prediction you want the model to learn.

Conclusion

Cross-entropy turns a classifier’s probability for the observed target into a useful training signal. For a hard target, the essential calculation is -log(p_y): correct and confident predictions have small loss, while confident mistakes are penalized sharply.

In neural classifiers, think of softmax and cross-entropy as one numerical operation built from logits rather than two unrelated steps. The target probability depends on every class score, and the resulting gradient raises the correct logit while lowering competing logits according to their current probabilities.

When implementing or interpreting the loss, keep the task structure in view. Distinguish multiclass from multilabel prediction, know whether the API expects logits, check reduction and weighting, and do not confuse a lower training loss with better generalization or calibrated confidence. With those details in place, cross-entropy becomes more than a formula: it becomes a clear model of what the classifier is being asked to learn.