Embedding vectors can occupy a narrow cone instead of spreading evenly across their available dimensions. In that geometry, unrelated items may still have noticeably positive cosine similarity because many vectors share a common directional component. This concentration is called anisotropy.
For developers, anisotropy matters at the point where vector geometry becomes an application signal. A similarity threshold, nearest-neighbor ranking, clustering rule, or novelty detector inherits the distribution produced by the embedding model. The same cosine value can carry different meaning across representation spaces with different directional concentration.
Cosine similarity can inherit a shared direction
Cosine similarity compares the angle between two nonzero vectors:
cos(a, b) = (a · b) / (||a|| ||b||)L2 normalization removes magnitude from this expression, but it does not force vectors to become directionally uniform. If many normalized vectors point partly along the same dominant direction, their pairwise dot products retain that shared component.
Consider normalized vectors decomposed conceptually into a common component and item-specific variation:
x_i ≈ c + r_iHere, c represents a direction shared across many items and r_i represents variation around it. This is not an assertion that a real embedding model uses an explicit additive decomposition. It is a useful geometric model for reasoning about a cloud with a dominant direction.
When c contributes strongly, cosine scores can be shifted upward across many pairs. A threshold copied from another embedding model then has no stable interpretation. Even a threshold retained across two versions of the same system should be rechecked if the representation distribution changes.
Anisotropy is a distribution property
A single vector cannot establish that an embedding space is anisotropic. The signal appears across a collection of vectors.
One simple diagnostic starts with L2-normalized embeddings and examines pairwise cosine similarities for pairs sampled without using relevance labels. A distribution concentrated far from the behavior expected for broadly dispersed directions suggests that the vectors share structure beyond pair-specific semantic relations. The exact baseline depends on dimensionality and the data distribution, so this diagnostic is comparative rather than a universal pass-fail test.
The mean vector is another useful signal:
mu = (1 / n) * sum(x_i)For normalized vectors, a large norm of mu indicates that directions do not cancel strongly across the sample. That can expose a shared orientation, though it does not describe all forms of anisotropy.
Principal-component analysis adds more detail. If a small number of directions account for a large share of variation, the spectrum of the covariance matrix will be uneven. An uneven spectrum alone is not a defect: real semantic data can contain genuine dominant factors. The practical question is whether those directions distort the similarity behavior required by the application.
Centering changes the score geometry
A common diagnostic transformation subtracts the sample mean:
z_i = x_i - muIf cosine similarity is used afterward, z_i is typically normalized again before comparison. This operation changes angles, not just vector offsets. It therefore changes nearest-neighbor rankings and score distributions.
That distinction is useful. If centering substantially separates score distributions that were previously compressed together, the original mean direction was contributing to the similarity signal. If little changes, another geometric property may dominate.
Centering is not automatically a suitable production fix. The estimated mean depends on the corpus or reference sample used to compute it. A mean estimated on one data distribution can become stale after a domain shift. Query vectors and indexed vectors also need compatible processing; subtracting a corpus mean from only one side changes the geometry asymmetrically.
The transformation should therefore be treated as part of the representation pipeline, with its parameters versioned alongside the embedding model and index.
Removing dominant components is a stronger intervention
Another family of transformations projects vectors away from one or more dominant directions. Given a unit direction u, removing its component from vector x can be written as:
x' = x - (x · u)uApplying this to principal directions can reduce concentration associated with those components. It also discards information. A dominant component may encode corpus structure that is useful for some queries, so removing it based only on its statistical prominence can reduce task quality.
The number of removed directions is therefore not a purely geometric tuning parameter. It changes the representation’s information content. Evaluation should compare application-level relevance or clustering behavior before and after the transformation, not merely seek a flatter component spectrum.
This also separates anisotropy correction from dimensionality reduction. Both can involve principal components, but their objectives differ. Dimensionality reduction commonly retains directions that explain substantial variance. A component-removal scheme aimed at directional concentration may discard some of those same directions. The operation only makes sense in relation to the downstream similarity objective.
Thresholds need calibration on the transformed space
Similarity thresholds are especially sensitive to post-processing. Suppose an application accepts a candidate when cosine similarity exceeds t. Centering, component removal, or switching embedding models can move both relevant and irrelevant score distributions. Keeping the old t assumes those distributions remain comparable, which the transformation does not guarantee.
A defensible evaluation keeps the decision rule visible. Measure scores for representative positive and negative pairs, apply the candidate transformation consistently, then inspect how the operating point changes. If labels are incomplete, retrieval metrics on a judged query set can still show whether ranking quality moved in the intended direction.
Nearest-neighbor systems also need separate checks for ranking and absolute scores. A transformation can preserve many top neighbors yet move their cosine values enough to break a threshold-based filter. Conversely, score ranges can look similar even when local rankings change.
Anisotropy and hubness can interact without being identical
Directional concentration and hubness describe different observations. Anisotropy concerns how vectors occupy directions in the representation space. Hubness concerns how often particular vectors occur in nearest-neighbor lists.
A concentrated space can contribute to uneven neighbor behavior, but observing one does not establish the other. Hub frequency should be measured from neighbor lists. Anisotropy should be examined through directional and similarity distributions. Keeping the diagnostics separate prevents a geometric label from replacing evidence about actual retrieval results.
The same separation applies to approximate nearest-neighbor indexing. An index can introduce recall error relative to exact search, but it does not create the original embedding distribution. Testing a manageable sample with exact similarity calculations helps isolate representation geometry from index approximation.
The reference distribution is part of the measurement
Anisotropy measurements depend on which vectors are sampled. A corpus containing one narrow domain can naturally occupy a smaller semantic region than a broad multi-domain corpus. Query embeddings can also have a different distribution from document embeddings, especially when the model or prompting scheme assigns distinct roles to each side.
For that reason, geometry checks should identify the population being measured. Corpus vectors answer questions about the indexed collection. Query vectors describe incoming traffic. Pairwise query-document scores describe the relation used by retrieval. Combining these into one undifferentiated statistic can hide the source of concentration.
Representation geometry is most actionable when tied to a concrete decision boundary. If cosine thresholds drift, irrelevant pairs cluster at unexpectedly high scores, or rankings become dominated by a narrow region, anisotropy measurements can show whether shared directions are part of the cause. Any correction then becomes a versioned change to the scoring space, not a cosmetic cleanup of the vectors.