A text embedding model turns text into a vector so that software can compare meaning numerically. The difficult part is not producing vectors. A neural network can produce vectors for almost any input. The difficult part is teaching the geometry of those vectors so that distances correspond to the relationships your application cares about.

Contrastive learning provides a practical way to do that. Instead of asking a model to predict a class label, you show it examples that should be close together and examples that should be farther apart. Training adjusts the encoder so that those relationships become easier to recover from the resulting vectors.

This article builds contrastive learning from a small text-retrieval example, explains a common softmax-based objective, and shows why negative examples, batch construction, similarity functions, and evaluation choices can matter as much as the loss formula itself.

Start with the relationship you want the vectors to preserve

Suppose you are building semantic search for a developer support site. A user searches for:

reset a forgotten password

A useful embedding model should place that query near a relevant document such as:

How to recover access when you cannot remember your password

and farther from an unrelated document such as:

How to export monthly billing invoices

Call the query the anchor. The relevant document is a positive example because the application wants the pair to be similar. The unrelated document is a negative example because the application wants it to rank below the positive.

The basic training signal is therefore relational:

anchor:   reset a forgotten password
positive: recover access when you cannot remember your password
negative: export monthly billing invoices

wanted geometry:
similarity(anchor, positive) > similarity(anchor, negative)

This is the central mental model. Contrastive learning does not need to assign an absolute meaning to each coordinate. It needs to shape the space so that useful relationships are reflected by similarity.

Encode each text, then compare the vectors

Let an encoder f map text to a vector:

q = f(query)
p = f(positive document)
n = f(negative document)

The training objective needs a similarity function. Cosine similarity is common for text embeddings:

cosine(a, b) = (a · b) / (||a|| ||b||)

If vectors are normalized to unit length before comparison, their norms are both 1, so cosine similarity reduces to a dot product:

normalize(a) · normalize(b) = cosine(a, b)

That equivalence is useful because it makes the scoring rule explicit. If training uses normalized vectors and cosine-like scores, but production retrieval uses raw dot products on unnormalized vectors, vector magnitude can change rankings. Training and retrieval do not have to use identical implementations, but their scoring semantics should be intentionally compatible.

For a simple teaching example, imagine the model currently produces these cosine similarities:

similarity(query, positive) = 0.42
similarity(query, negative) = 0.35

The positive is ahead, but only slightly. Contrastive training should create pressure to increase the positive’s relative score and decrease the negative’s relative competitiveness.

Turn similarities into a contrastive objective

A common family of objectives treats the positive as the correct choice among several candidates. For one anchor with one positive and K negatives, define candidate scores s_0, s_1, ..., s_K, where s_0 is the positive score.

A softmax-style contrastive loss can be written as:

loss = -log(
  exp(s_positive / T)
  / sum(exp(s_candidate / T) for every candidate)
)

T is a positive temperature. It rescales the logits before the softmax. Lower temperatures make score differences produce sharper softmax probabilities; higher temperatures make the distribution flatter. Temperature therefore changes the optimization pressure and is a training hyperparameter, not a universal constant.

Consider three candidates with similarities:

positive document:  0.80
negative A:          0.50
negative B:          0.10

With T = 0.1, the scaled logits are:

8.0, 5.0, 1.0

The positive receives most of the softmax probability, so its loss is relatively small. If negative A instead scores 0.79, the model must distinguish two nearly tied candidates and the loss becomes much larger.

This illustrates an important property: a negative that is already obviously unrelated contributes less pressure than a negative that competes strongly with the positive.

The exact objective varies across systems. Some losses compare pairs, some compare triplets, and some use every other example in a batch as a candidate. The reusable idea is the same: reward the geometry that makes desired relationships easier to distinguish.

Use the batch to create more comparisons

Explicitly encoding many negatives for every anchor can be expensive. A common strategy is to construct a batch from matched pairs:

(q1, d1)
(q2, d2)
(q3, d3)
(q4, d4)

where di is the positive document for qi.

After encoding the four queries and four documents, compute a similarity matrix:

          d1    d2    d3    d4
q1       0.82  0.21  0.34  0.10
q2       0.18  0.76  0.29  0.33
q3       0.40  0.25  0.71  0.20
q4       0.12  0.38  0.22  0.79

The diagonal contains the intended positive pairs. For q1, the other documents can act as negatives; the same is true for the other rows. This technique is often called using in-batch negatives.

It is computationally attractive because the documents are encoded once and then reused across many comparisons. Increasing the batch can also expose each anchor to more negatives.

But a larger batch is not automatically a better training set. The additional candidates must be valid negatives for the task.

False negatives can teach the wrong geometry

Suppose the batch contains these two documents:

d1: Reset your password from the account recovery page
d2: Recover an account when you forgot the password

If d1 is labeled as the positive for a password-reset query while d2 belongs to another pair, a naive in-batch objective may treat d2 as a negative. Semantically, however, both documents may be valid answers.

That creates a false negative: training tells the model to separate items that the application would accept as related.

False negatives are especially likely when:

  • multiple documents answer the same question;
  • paraphrases appear in different pairs;
  • product catalogs contain near-duplicates;
  • labels capture only one known positive even though several positives exist.

The problem is not merely noisy bookkeeping. The loss directly converts the assumed negative relationship into gradient pressure.

Possible responses depend on the data. You can remove known equivalent items from the negative set, represent multiple positives explicitly, group duplicates before batching, or design batches so that ambiguous candidates are not automatically treated as negatives. None of these approaches can recover relationships that the dataset does not reveal, so data inspection remains important.

Easy negatives and hard negatives teach different things

Imagine a password-reset query with two possible negatives:

easy negative:
How to configure dark mode

hard negative:
How to change your password after signing in

The easy negative differs in both topic and vocabulary. A reasonably trained model may already place it far away. The hard negative is about passwords but answers a different intent. Distinguishing it can teach a boundary that matters for search quality.

A hard negative is a non-relevant candidate that the current model or another retrieval system considers similar to the anchor. Hard-negative mining often works by retrieving high-scoring candidates and keeping those known not to be relevant.

Hard negatives can make training more informative, but they also increase the cost of labeling mistakes. A supposedly non-relevant high-scoring document may actually be another valid answer. Aggressive hard-negative mining without reliable relevance judgments can therefore push useful neighbors apart.

A practical progression is:

start with trustworthy pairs and ordinary negatives
        ↓
train a reasonable baseline
        ↓
inspect retrieval errors
        ↓
add verified hard negatives that represent real confusions

This keeps difficulty tied to observed failure modes rather than assuming that the hardest-looking candidate is always the best negative.

The data definition determines what similarity means

There is no single semantic geometry that is correct for every application.

Consider the sentence:

Python list append is slow in this loop

For general semantic similarity, a paragraph explaining Python list performance may be a strong positive. For duplicate-bug detection, only reports describing the same underlying defect should be positive. For question-answer retrieval, a document must help answer the question, not merely discuss the same subject.

Those tasks can assign different relationships to the same pair of texts.

Contrastive training therefore begins with a product decision: what should count as similar? The model learns from the operational definition encoded by positive and negative examples.

This is why simply collecting paraphrases is not sufficient for every embedding application. If production needs query-to-document relevance, train and evaluate relationships that resemble query-to-document relevance.

Decide whether the encoder is symmetric

Some embedding tasks are naturally symmetric. In duplicate detection, either sentence can play either side of the comparison:

A: The build fails after upgrading the compiler
B: Compiler upgrade causes the build to fail

Other tasks are asymmetric. Search compares a short query with a document that may be much longer:

query -> document

You can still use one shared encoder for both sides, but sharing parameters is an architectural choice, not a mathematical requirement of contrastive learning. Some systems use different encoders or different input prefixes for queries and documents because the two roles carry different distributions or instructions.

The important point is to preserve the same role assumptions at training and inference time. If training marks inputs as queries and documents in distinct ways, production should not silently discard that distinction.

Do not judge the model by training loss alone

A lower contrastive loss means the model is fitting the comparisons represented by the training objective more successfully. It does not guarantee better application behavior.

For retrieval, evaluate on held-out queries with relevance judgments. Depending on the product, useful metrics may include recall at a cutoff, ranking metrics, or task-specific success rates. The exact metric should match what users need.

For example, if a downstream RAG system sends the top five retrieved chunks to a language model, Recall@5 may answer an important question: how often does at least one relevant item appear in the candidate set? If only the first result is shown to a user, quality at rank one deserves more weight.

Also evaluate slices that can expose hidden failures:

short queries vs long queries
common topics vs rare topics
new terminology vs familiar terminology
near-duplicate documents vs clearly distinct documents

A single average metric can improve while an important subgroup gets worse.

Watch the practical cost of stronger training

Contrastive training creates several cost-quality trade-offs.

Larger batches can provide more in-batch candidates, but they require more memory for representations and may increase compute and communication in distributed training. They can also introduce more false negatives if batch construction is careless.

Hard-negative mining can improve the relevance of training comparisons, but retrieving, storing, filtering, and refreshing mined examples adds pipeline complexity. Negatives mined by an old model may also become too easy as training progresses.

Longer text inputs may preserve more evidence, but encoder cost generally grows with sequence length and model architecture. Truncation can remove the passage that makes a document relevant. Before increasing maximum length, inspect where relevant information appears and whether chunking or task-specific preprocessing is more appropriate.

Finally, embedding dimensionality affects storage and retrieval cost in production. Contrastive learning can improve the representation at a chosen dimension, but it does not make vector size free. Model quality should be evaluated together with index size, retrieval latency, and serving cost.

Common mistakes to avoid

Treating every other batch item as a guaranteed negative

In-batch negatives are labels created by your batching assumption. If two examples are genuinely related, the loss can receive a contradictory signal. Check for duplicates, multiple valid answers, and other sources of semantic overlap.

Mining negatives only because they score highly

A high score tells you that a retriever finds an item similar. It does not prove irrelevance. Hard negatives are most useful when relevance is known with enough confidence to justify pushing the pair apart.

Changing similarity rules between training and serving

If training normalizes vectors for cosine-style scoring but serving uses raw dot products, vector norms can alter rankings. Document the representation and scoring contract and test it end to end.

Optimizing a proxy that does not match the product

Training on generic sentence similarity may not produce the ranking behavior needed for technical support search. Build positives, negatives, and evaluation sets around the relationship the application actually consumes.

Reading embedding distance as calibrated confidence

A similarity score is useful for ranking, but it is not automatically a probability that two texts are relevant. Thresholds should be validated on representative held-out data if the application turns similarity into a yes/no decision.

When contrastive learning is a good fit

Contrastive learning is especially useful when the desired output is a reusable representation and supervision naturally comes as relationships: query-document relevance, duplicate pairs, paraphrases, matching products, or other notions of relatedness.

It is less compelling when the task is already solved well by a direct classifier and you do not need reusable vectors or nearest-neighbor retrieval. A classifier can learn a decision boundary without requiring you to design an embedding space for general comparison.

It may also be premature when positive and negative relationships are unreliable. In that case, improving labels and evaluation data can be more valuable than changing the loss function. Contrastive objectives are effective at enforcing the relationships you provide, including incorrect ones.

Conclusion

Contrastive learning is easiest to understand as geometry shaped by comparisons. Positive examples define which representations should move together, negatives define which distinctions matter, and the loss converts those relationships into training pressure.

For developers, the most important choices are often outside the formula. Define similarity according to the product, keep training and serving similarity rules compatible, treat in-batch and hard negatives as data assumptions rather than free supervision, and evaluate retrieval behavior on held-out examples that resemble production. When those pieces align, contrastive learning provides a practical foundation for text embeddings that encode the relationships your system actually needs.

Artificial Intelligence Machine Learning Embeddings Training