Understand Neural Collapse in Deep Classifiers

A classifier can keep changing after it already predicts every training example correctly. Cross-entropy loss can continue to fall, feature vectors can reorganize, and the final classification layer can become increasingly regular. Looking only at training accuracy hides all of that movement.

Neural collapse is a name for a collection of geometric patterns that can emerge late in the training of deep classifiers. The striking part isn’t simply that examples from the same class become similar. Under the conditions where neural collapse appears, within-class variation can shrink while class centers and classifier weights approach a highly symmetric arrangement.

This article develops that picture from a small feature-space example. You’ll see what is actually collapsing, how the main pieces fit together, how to measure the pattern, and why observing neural collapse is a diagnostic rather than proof that a model will generalize well.

Start at the layer before the classifier

Consider a neural network trained to classify images into three classes: cat, dog, and bird. Near the end of the network, suppose each input is represented by a feature vector h(x). A final linear layer turns that vector into class logits:

logits = W h(x) + b

Each row of W corresponds to one class. The network predicts the class with the largest resulting logit.

Early in training, feature vectors from the same class may be scattered. A two-dimensional teaching picture might look like this:

cat:   (1.0, 2.1)  (2.0, 1.4)  (0.8, 1.2)
dog:   (-1.8, 0.5) (-0.7, 1.6) (-1.1, -0.2)
bird:  (0.2, -1.5) (1.1, -0.8) (-0.6, -2.0)

The classes may already be separable, but examples within a class still occupy noticeably different positions.

Now imagine continued optimization causes the cat features to concentrate near one point, the dog features near another, and the bird features near a third. The individual training examples haven’t become identical as inputs. Their learned representations have become much more similar within each class.

That distinction matters. Neural collapse describes geometry in a learned feature space and in the classifier built on top of it. It does not mean the original data loses its variation.

What actually collapses

Neural collapse is usually discussed as several related properties rather than one scalar event. The exact behavior depends on the training setup, but the core picture has four parts.

Within-class features concentrate

For class c, let its feature vectors be h_1, h_2, ..., h_n. Define the class mean

mu_c = average feature vector for class c

A simple measure of within-class spread is the average squared distance from each feature to its class mean:

within_class_variance(c)
    = average ||h_i - mu_c||^2

The first neural-collapse pattern is that this within-class variability becomes small relative to the separation between classes. Training examples from one class increasingly cluster around the same class mean.

This is the most literal sense of the word collapse: many distinct feature vectors approach a common class-specific representation.

Centered class means become highly symmetric

The class means themselves don’t collapse onto one point. They move apart in a structured way.

Let mu_G be the global mean of the class means, and define centered class means

m_c = mu_c - mu_G

In the idealized neural-collapse geometry, these centered vectors have equal norms and equal pairwise angles. For C classes, their normalized pairwise inner products approach

-1 / (C - 1)

for distinct classes, provided the feature space has enough dimensions to realize this arrangement.

For three classes, the value is -1/2, corresponding to angles of 120 degrees. In two dimensions, you can picture the centered class means as three equally spaced spokes:

                 cat
                  *
                 / \
                /   \
               /     \
            *-----------*
          dog           bird

For four classes, the analogous ideal geometry is a regular tetrahedral arrangement in three dimensions. In general, the centered means form what is called a simplex equiangular tight frame. The name is less useful than the mental model: equal-length class directions spread as evenly as possible around their shared center.

Classifier weights align with class directions

The final linear classifier also becomes tied to this geometry. Its class weight vectors tend to align with the centered class means, up to scaling under the idealized pattern.

That makes the classifier easier to visualize. Instead of learning an arbitrary decision rule over a complicated cloud of training features, the late-stage system approaches something closer to a set of class prototypes with matching classifier directions.

The alignment doesn’t mean the final layer is literally storing a training example. The prototype is a direction derived from the learned representations of an entire class.

Classification approaches nearest class center

Once within-class spread is small, class means are well separated, and classifier weights align with those means, the network’s decision becomes closely related to choosing the nearest class center in feature space.

This final property follows naturally from the earlier geometry. If each class occupies a tight cluster and the classifier points along the corresponding class direction, both views of the decision boundary become similar.

This is useful as a mental model, but it shouldn’t be promoted into an implementation guarantee. A real network can show some neural-collapse properties more strongly than others, and finite training runs need not reach the ideal geometry.

Why training can continue after accuracy reaches 100 percent

Suppose every training example is already classified correctly. For one example, the logits might be:

cat:   4.0
dog:   1.0
bird: -0.5

The prediction is correct, but ordinary cross-entropy loss is not zero. The softmax still assigns some probability to the wrong classes. Increasing the margin between the correct logit and the others can reduce the loss further.

So zero training error doesn’t imply that optimization has stopped having an objective. The model can continue changing its weights and features while preserving the same predicted labels.

That late optimization is the setting in which neural-collapse behavior is often studied. The distinction between classification error and cross-entropy loss explains why looking only at accuracy can miss substantial changes in representation geometry.

It also explains why neural collapse isn’t simply another name for overfitting. Overfitting describes a gap between performance on training data and unseen data. Neural collapse describes a particular geometric organization of learned representations and classifier weights. The two questions can interact, but they are not the same question.

Measure the geometry instead of relying on a plot

Two-dimensional projections are useful for intuition, but they can distort angles and distances. If you want to check for neural-collapse behavior, calculate quantities in the original feature space.

Assume you collect penultimate-layer features and labels for a fixed evaluation set. A practical diagnostic can track three families of measurements across checkpoints.

First, compare within-class spread with between-class scale. One simple ratio is

sum of within-class squared distances
-------------------------------------
sum of squared centered class means

The exact normalization can vary, so use one definition consistently. A decreasing ratio indicates that examples are becoming tighter around their class centers relative to the separation of those centers.

Second, normalize each centered class mean and build their Gram matrix:

G[i, j] = normalized(m_i) dot normalized(m_j)

For an ideal C-class simplex, the diagonal entries are 1, while off-diagonal entries are -1/(C-1). With three classes, the target matrix is

[ 1.0  -0.5  -0.5 ]
[-0.5   1.0  -0.5 ]
[-0.5  -0.5   1.0 ]

You can compare the measured matrix with that pattern. This tests symmetry directly rather than asking whether a scatter plot “looks triangular.”

Third, measure alignment between each normalized classifier weight and its corresponding normalized centered class mean. Cosine similarity is a natural choice:

alignment(c) = cosine(w_c, m_c)

Values moving toward 1 indicate increasing directional alignment. Be careful with biases and centering conventions: if your analysis assumes a centered classifier but your model uses a learned bias, account for that rather than silently dropping it.

These diagnostics are most informative as trajectories. A single checkpoint may show a moderately symmetric geometry for many reasons. Watching within-class spread, class-mean geometry, and weight alignment evolve together gives stronger evidence that you’re observing the coupled phenomenon.

A small implementation pattern

You don’t need a special training algorithm to inspect neural collapse. The main requirement is access to features from the layer immediately before the final classifier.

A framework-neutral procedure looks like this:

for each checkpoint:
    features = run examples through model up to penultimate layer

    for each class c:
        mu[c] = mean(features where label == c)

    global_mean = mean(mu over classes)
    centered[c] = mu[c] - global_mean

    measure within-class spread
    measure pairwise cosine similarities of centered means
    measure cosine alignment between classifier weights and centered means

For a production diagnostic, decide which data split you are measuring. Neural collapse is commonly analyzed on training representations, because that is where the optimization geometry is most directly visible. Measuring validation features can answer a different and useful question: whether unseen examples inherit a similar class structure. Don’t combine those results without labeling them.

Class balance also deserves attention. The clean symmetric picture is most naturally associated with balanced classes and symmetric treatment of classes. Strong imbalance, class weighting, label noise, regularization choices, or architectural constraints can change the geometry. If your training objective deliberately treats classes differently, a perfectly symmetric simplex may not even be a sensible expectation.

What neural collapse does not tell you

The regularity of neural collapse makes it tempting to treat it as a quality score. That is too strong.

A model can organize its training features neatly and still fail on distribution shift. If all training images of bird share a background artifact, a network may learn a compact bird representation around that shortcut. Tight clustering doesn’t tell you that the representation uses the right evidence.

Likewise, a model that doesn’t approach the textbook geometry isn’t automatically defective. The assumptions behind the ideal picture may not match the task. Multi-label classification, hierarchical labels, severe class imbalance, metric-learning objectives, or models without a standard linear classification head can require different geometry.

Neural collapse also shouldn’t replace ordinary evaluation. Accuracy, class-specific error rates, calibration, robustness tests, and task-specific costs answer operational questions that feature geometry alone cannot answer.

The most defensible use is diagnostic: neural collapse can reveal how a classifier’s internal representation is organizing itself late in training. It can help distinguish “the labels are already correct” from “the representation has stopped changing,” which are very different statements.

Common mistakes when interpreting neural collapse

One mistake is measuring raw class means without centering them. The symmetric relationship concerns class means relative to their global center. A shared offset can obscure the relevant angles.

Another is checking only within-class variance. Tight clusters are part of neural collapse, but ordinary representation learning can also produce tight clusters. The relationship among class centers and classifier weights is what makes the full pattern more specific.

A third mistake is declaring success from a two-dimensional visualization. PCA or another projection can be useful for communication, but it discards information. Compute the diagnostics in the original feature dimension and use projections only as supporting views.

Finally, don’t optimize directly for a neural-collapse metric unless you have a task-specific reason and have validated the consequence you care about. A prettier Gram matrix isn’t itself a product requirement. If the real goal is better accuracy on rare classes, lower false positives, or stronger out-of-distribution behavior, evaluate that goal directly.

When this mental model is useful

Neural collapse is most useful when you’re trying to understand late-stage behavior in a conventional deep classifier: a learned feature extractor, a linear classification head, mutually exclusive classes, and cross-entropy-style training. It gives you a vocabulary for changes that continue after training accuracy has saturated.

It can also help when comparing checkpoints. Two checkpoints with identical training accuracy may have quite different feature spread, class-center geometry, and classifier alignment. Those measurements expose structure that the accuracy number discards.

For many application decisions, though, simpler diagnostics should come first. If your model is underfitting, labels are noisy, validation accuracy is falling, or one class has poor recall, investigate those concrete problems before studying asymptotic feature geometry. Neural collapse is a lens on representation structure, not a substitute for debugging the training pipeline.

Use geometry as evidence, not a verdict

The practical lesson from neural collapse is that a classifier can become much more organized even after its training predictions stop changing. Within-class features can concentrate, centered class means can approach a symmetric simplex, and classifier weights can align with those class directions.

If you want to inspect that process, capture penultimate-layer features across checkpoints and measure the geometry directly. Then keep the interpretation narrow: the measurements tell you how the representation is organized. Whether that organization is useful still has to be answered on validation data and on the real failures your application cares about.