Cosine similarity is often treated as if a score has the same meaning across any embedding space. That assumption breaks when vectors occupy a narrow region of the available geometry. If many embeddings share a strong common direction, unrelated items can receive positive cosine scores simply because both align with that direction.

This behavior is usually described as embedding anisotropy. It is not a defect in cosine similarity itself. The issue is that cosine measures angles in the representation it receives, including global structure that may have little value for the downstream comparison.

Isotropy is a property of a set of vectors

An embedding vector cannot be anisotropic by itself. The term describes how a collection of vectors is distributed across directions.

In an idealized isotropic cloud, directions are comparatively balanced. In an anisotropic cloud, variance or mean direction is concentrated along a smaller set of axes. Real embedding spaces can sit anywhere between those extremes.

A simple two-dimensional picture makes the effect visible. Suppose vectors are:

a = [10, 1]
b = [10, 2]
c = [10, -1]

All three share a large first component. Their cosine similarities are high because the common component dominates their norms and dot products. Differences in the second component contribute much less to the final angle.

The same principle extends to higher dimensions. A shared component can come from the representation model, input distribution, pooling rule, or another systematic property of the pipeline.

A nonzero mean is one useful diagnostic

Given embeddings x_1 ... x_n, their empirical mean is:

mu = (1 / n) * sum(x_i)

A large mean vector relative to typical embedding norms indicates that the sampled vectors share a directional component. This is not a complete test for anisotropy, since a zero-mean distribution can still have variance concentrated in a few directions, but it is an inexpensive signal.

Pairwise cosine statistics add another view. If random pairs from a representative corpus cluster around a substantial positive similarity rather than spreading around a lower baseline, shared geometry may be influencing scores.

The sampling distribution matters. Measurements from source code, support tickets, and product descriptions can differ even with the same encoder. A geometric diagnostic is therefore tied to both the model and the data being embedded.

Centering removes the empirical mean direction

Mean centering transforms each vector as:

x_centered = x - mu

Cosine similarity can then be computed using the centered vectors. This removes the mean of the reference set exactly for that set:

(1 / n) * sum(x_i - mu) = 0

Centering can increase the relative influence of dimensions that were previously overshadowed by a shared offset. It also changes every vector norm and angle, so centered cosine scores are not interchangeable with scores from the original space.

That last point has practical consequences for vector indexes. If stored document vectors are centered, query vectors must be transformed with the same reference mean before comparison. Mixing centered and uncentered vectors produces a geometry that has no consistent interpretation.

The mean should also come from an appropriate reference population. Recomputing it independently for each query batch would make the coordinate system move from request to request.

Principal directions reveal concentration beyond the mean

Centering addresses a shared offset but does not make a distribution isotropic. After centering, variance can remain heavily concentrated along a few directions.

A covariance matrix summarizes that structure:

C = (1 / n) * sum((x_i - mu) (x_i - mu)^T)

Its eigenvectors define principal directions, and the corresponding eigenvalues describe variance along those directions. If a small number of eigenvalues dominate, much of the observed variation lies in a low-dimensional subspace.

This view distinguishes two cases that a mean check alone cannot separate. A vector set can have a large common mean, concentrated covariance, both, or neither.

For very high-dimensional embeddings, developers do not need a full dense covariance decomposition merely to inspect the issue. Approximate singular-value methods or a representative matrix sample can expose dominant directions, provided the approximation and sample are adequate for the diagnostic being made.

Removing dominant components is a stronger intervention

A more aggressive transform can subtract projections onto selected principal directions. For a unit direction u, removing its component from vector x is:

x' = x - (x dot u) * u

Applying this to several directions suppresses variation associated with those axes.

This operation should not be treated as a generic improvement. A dominant direction can encode nuisance structure, useful semantics, or a mixture of both. Removing it discards information by construction. The effect must be judged against the actual retrieval, clustering, or classification objective.

The number of removed directions is therefore a model-selection choice, not a universal constant. It should be fixed using reference data that is separate from the final evaluation set.

Whitening changes scale as well as direction

Whitening goes further by rescaling centered principal components according to their variance. In an idealized full-rank case, a whitening transform produces covariance close to the identity on the data used to fit it.

That can make low-variance directions contribute more strongly and high-variance directions contribute less strongly. Such a transformation is substantially different from simple centering. It changes the metric structure of the space, not just its origin.

Small eigenvalues also require care. Dividing by the square root of a value near zero can amplify numerical noise. Practical whitening commonly needs dimensionality reduction, regularization, or a floor on the scale factor.

As with centering, the fitted transform becomes part of the embedding pipeline. Stored vectors and incoming queries must use the same transform.

Score distributions matter more than isolated examples

An individual pair with cosine similarity 0.8 says little without context. If random corpus pairs usually score near 0.2, that value occupies a different part of the score distribution than it would in a space where random pairs commonly score near 0.75.

For retrieval systems, useful diagnostics include the distribution of random-pair scores, nearest-neighbor score gaps, and task metrics before and after any geometric transform. These measurements reveal whether anisotropy is merely visible in the embedding cloud or is actually interfering with ranking.

A transform can make a histogram look more balanced while degrading retrieval. Geometry is a means to inspect and modify representation behavior; downstream evaluation remains the deciding test.

Reference data becomes part of the representation contract

Centering, principal-component removal, and whitening are fitted transformations. Once deployed, their fitted parameters are as much a part of the representation contract as the encoder version and pooling method.

Changing the reference corpus can change the mean, principal directions, and scales. Re-embedding only part of an index with newly fitted parameters can therefore create incompatible vectors even when the base encoder has not changed.

Embedding anisotropy is most useful as a diagnostic concept: it explains how global geometry can shift cosine scores away from pair-specific semantic alignment. Correcting that geometry can help under some data distributions, but the correction itself changes the representation. Treating its parameters as versioned model artifacts keeps that change explicit and testable.

Artificial Intelligence Embeddings Vector Search Representation