Use Cosine Classifiers to Separate Direction from Feature Magnitude

A standard neural classifier usually ends with a linear layer. That head scores each class with a dot product between the model’s feature vector and a class weight vector. The dot product mixes two signals: the angle between the vectors and their magnitudes.

A cosine classifier removes the magnitude part before scoring. It normalizes both vectors to unit length, then compares their directions. An explicit scale converts the resulting cosine similarities into logits suitable for a classification loss.

This small architectural change gives developers a useful control: class decisions can depend on feature direction while logit magnitude is handled separately. That can be useful when feature norms vary for reasons that should not directly decide the class.

A linear head mixes angle and magnitude

Suppose an encoder maps an input to a feature vector (h \in \mathbb{R}^d). An ordinary bias-free linear classifier has one weight vector (w_k) for each class (k):

[ z_k = w_k^\top h ]

where (z_k) is the logit for class (k).

The dot product can also be written as:

[ w_k^\top h

|w_k|_2 , |h|_2 \cos(\theta_k) ]

Here, (\theta_k) is the angle between the feature and class weight. A large logit can therefore come from close directional alignment, a large feature norm, a large class-weight norm, or some combination of all three.

That flexibility is often useful. It also means the geometry can be harder to interpret. Two inputs pointing in exactly the same direction can receive logits of very different magnitude if their feature norms differ.

Consider two feature vectors:

[ h_a = [3,4], \qquad h_b = [0.6,0.8] ]

They point in the same direction because (h_a = 5h_b). With a class weight (w=[1,0]), the ordinary dot products are:

[ w^\top h_a = 3 ]

and

[ w^\top h_b = 0.6 ]

The head treats the vectors differently even though their directions match.

A cosine classifier normalizes before scoring

A cosine classifier replaces each vector with its unit-length version:

[ \hat{h} = \frac{h}{|h|_2} ]

[ \hat{w}_k = \frac{w_k}{|w_k|_2} ]

The class score becomes:

[ c_k = \hat{w}_k^\top \hat{h} ]

For nonzero vectors, this is the cosine of the angle between them, so (c_k) lies between (-1) and (1).

The two features from the earlier example both normalize to:

[ [0.6,0.8] ]

Their cosine score against the normalized class vector ([1,0]) is therefore (0.6) in both cases.

This changes the role of the classifier weights. Their directions define class prototypes in feature space, while their raw norms no longer affect the forward score.

A practical implementation must handle very small norms. Normalization functions typically use an epsilon or another lower bound in the denominator to avoid division by zero. The exact numerical convention is framework-dependent, so it belongs in the implementation contract rather than being assumed from the formula alone.

Cosine values need an explicit logit scale

Using raw cosine similarities directly with softmax cross-entropy creates a restricted logit range. A score cannot exceed (1) or fall below (-1), so even strong angular separation may produce a relatively soft probability distribution.

For two classes with cosine scores (1) and (-1), the softmax probability of the first class is:

[ \frac{e^1}{e^1 + e^{-1}} \approx 0.881 ]

The geometry is maximally separated along those two directions, yet the probability is still below (0.9).

Cosine classifiers therefore commonly multiply similarities by a positive scale (s):

[ z_k = s , \hat{w}_k^\top \hat{h} ]

If (s=10), the same two scores become (10) and (-10), producing a much more concentrated softmax distribution.

The scale does not change which class has the largest cosine similarity when the same positive value is applied to every class. It does change the loss surface and gradient magnitudes produced through softmax. As a result, the scale affects training behavior rather than serving as a cosmetic post-processing constant.

Some systems keep (s) fixed. Others parameterize it and fit it during training. Either approach can work, but the scale should be treated as part of the model definition and evaluated rather than copied from an unrelated setup.

A small three-class example

Imagine a two-dimensional feature space with three normalized class vectors:

[ \hat{w}_A = [1,0] ]

[ \hat{w}_B = [0,1] ]

[ \hat{w}_C = \left[ -\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}} \right] ]

An input produces the normalized feature:

[ \hat{h} = [0.8,0.6] ]

The cosine similarities are:

[ c_A = 0.8 ]

[ c_B = 0.6 ]

[ c_C = -\frac{1.4}{\sqrt{2}} \approx -0.990 ]

Class (A) has the highest score because its direction is closest to the feature direction.

With scale (s=5), the logits become approximately:

[ [4.0,\ 3.0,\ -4.95] ]

The scale increases the separation passed to softmax, but it does not alter the angular ranking.

This example also exposes an important limitation. A two-dimensional unit circle has little room for many well-separated class directions. In real models, the feature dimension, number of classes, encoder capacity, data geometry, and objective all influence how much angular separation is achievable.

Bias changes the geometric interpretation

A standard linear layer often includes a bias:

[ z_k = w_k^\top h + b_k ]

A pure cosine classifier usually omits a per-class bias because adding (b_k) shifts the score independently of the angle. Once that happens, the decision is no longer determined solely by angular similarity.

That doesn’t make bias invalid. It means the model is no longer a pure directional classifier.

This distinction matters when the architecture is chosen specifically to make class scores interpretable as scaled cosine similarities. If a per-class offset is required for the task, an ordinary linear head or a deliberately modified cosine head may be a better description of the model.

Normalization changes the gradients too

Cosine normalization is not merely a different inference formula. During end-to-end training, gradients pass through the normalization operation.

For a feature vector (h), changing its raw magnitude without changing its direction does not directly change the normalized feature. The classification objective therefore puts its direct scoring pressure on direction rather than on making the feature norm arbitrarily large.

The encoder can still use magnitude internally before the final normalization point. Other losses can also depend on the unnormalized representation. The architectural statement is narrower: the cosine classification score itself discards the final feature magnitude.

Class weights behave similarly. Their directions matter to the forward score, while their raw magnitudes are removed by normalization. Optimizer state, weight decay, numerical precision, and the normalization operation can still interact with the underlying parameters during training, so raw weight norms are not operationally irrelevant.

Cosine classifiers can fit prototype-style decisions

A useful mental model is to view each normalized class vector as a direction representing that class. The encoder maps an input onto the same unit sphere, and classification chooses the class direction with the greatest similarity.

This geometry can fit tasks where relative direction is a meaningful representation and raw feature norm is an unwanted source of score variation. It also creates a natural connection between classification and embedding-based comparison: both can use normalized vectors and cosine similarity.

That does not mean a cosine head automatically produces a good embedding space. Cross-entropy still optimizes the classification objective supplied during training. If downstream embedding quality matters, evaluate retrieval, clustering, verification, or other relevant behavior directly.

A cosine classifier is also not a replacement for probability calibration. The scale can make softmax outputs more or less concentrated, and high softmax confidence does not by itself establish that predicted probabilities match empirical correctness rates. Calibration should be measured separately when probabilities drive decisions.

Compare against a linear head under controlled conditions

The cleanest evaluation keeps the encoder, training data, split, optimizer family, augmentation, and primary metric aligned while changing the head and any settings required by that head.

Track more than final accuracy. Depending on the application, useful checks include:

• validation loss and task metric;
• distribution of feature norms before normalization;
• angular separation between class vectors;
• per-class performance, especially for imbalanced data;
• calibration metrics when probability quality matters;
• behavior on shifted or difficult inputs.

Feature norms deserve special attention. If an ordinary linear head performs better, the norm may be carrying useful predictive information. Removing it is not automatically an improvement.

The reverse can also occur: a cosine head may make optimization or class geometry more suitable for a particular setup. Treat that as an empirical property of the model and data, not as a universal advantage of normalization.

Common implementation mistakes

The first mistake is normalizing along the wrong dimension. For a batch of features shaped [batch, dimension], each example should normally be normalized across its feature dimension. A class-weight matrix shaped [classes, dimension] should normally normalize each class vector across that same representation dimension.

The second mistake is forgetting the logit scale. Raw cosine values may give a weak training signal for a softmax objective, particularly when the desired class probabilities require larger logit gaps.

The third mistake is allowing the scale to become unconstrained if the implementation assumes it must remain positive. A trainable scale can be parameterized through a transformation that preserves the intended domain, or otherwise constrained according to the model design.

The fourth mistake is assuming normalization makes all numerical issues disappear. Near-zero vectors need stable handling, mixed-precision behavior can depend on kernels and dtypes, and normalization adds computation that may or may not be material in a given deployment.

The fifth mistake is comparing heads with different training budgets or tuning effort. A cosine head with a carefully tuned scale should not be declared superior to an untuned linear baseline.

Cases where a linear head is simpler

An ordinary linear classifier is a strong default. It has fewer geometric constraints and lets the model use both direction and magnitude.

Keep the linear head when feature magnitude is plausibly informative, when the existing model already meets requirements, or when there is no concrete benefit from enforcing directional scoring. Extra architectural constraints should earn their place through measurable improvements or clearer system behavior.

A cosine classifier becomes more compelling when you specifically want magnitude-invariant final scoring, a prototype-like angular geometry, or a close match between classification and normalized embedding comparisons.

Even then, the decision belongs in validation. Compare both heads on the outcomes the application actually values.

Treat direction and confidence as separate controls

The central idea of a cosine classifier is separation of responsibilities. Normalized features and class vectors determine angular compatibility. The scale determines how strongly those similarities are presented to the classification loss.

That separation can make the final layer easier to reason about: changing feature magnitude alone cannot increase a class score, and class-weight magnitude alone cannot make one class dominate. The model must improve directional alignment instead.

Use that constraint when it matches the representation you want. Keep a linear head when magnitude should remain part of the decision. In either case, measure task quality, calibration, and behavior on representative data rather than inferring model quality from the geometry alone.