A transformer usually produces one contextual representation for every input token. Many applications, however, need one vector for an entire sentence, query, or document. Semantic search, clustering, and similarity systems commonly compare these fixed-size vectors rather than every token representation separately.
The operation that turns a variable number of token vectors into one vector is called pooling. It can look like a minor implementation detail, but changing it changes the representation being compared. Averaging every meaningful token, selecting a designated token, or emphasizing particular positions encodes different assumptions about where useful information lives.
This article builds a practical mental model for pooling, works through a small example, and explains how to choose and evaluate a strategy without treating any pooling rule as universally superior.
Start with token representations, not one sentence vector
Suppose a transformer processes this short query:
reset database passwordAfter tokenization and encoding, imagine that three content tokens have these simplified two-dimensional representations:
reset -> [ 1.0, 0.2 ]
database -> [ 0.4, 1.4 ]
password -> [ 0.8, 1.0 ]Real transformer vectors usually have hundreds or thousands of dimensions, and tokenization may split words into multiple tokens. Two dimensions simply make the pooling arithmetic visible.
If a retrieval index expects one vector per query, these three vectors cannot be stored or compared as one ordinary dense embedding without an aggregation rule. Pooling supplies that rule.
The important distinction is:
transformer encoder: tokens -> contextual token representations
pooling: token representations -> fixed-size sequence representationPooling does not recover the original text or preserve every token-level detail. It compresses a sequence of vectors into a representation intended to be useful for a downstream objective.
Mean pooling gives every included token equal weight
The simplest common strategy is mean pooling. For n included token vectors h_1 ... h_n, compute each output dimension as their arithmetic mean:
pooled = (h_1 + h_2 + ... + h_n) / nFor the three-token example:
first dimension = (1.0 + 0.4 + 0.8) / 3 = 0.7333...
second dimension = (0.2 + 1.4 + 1.0) / 3 = 0.8666...
mean embedding ~= [0.733, 0.867]The mental model is straightforward: each included token contributes equally to the final vector.
That simplicity is useful when meaning is distributed across the sequence. It also creates a trade-off. A highly informative token receives no more direct weight than a weakly informative token unless the transformer’s contextualization has already encoded that importance into the token vectors themselves.
Mean pooling is therefore an aggregation rule, not a guarantee that every token deserves equal semantic importance.
Exclude padding from the mean
Batched inputs are often padded to a common sequence length. Padding exists to make tensor shapes convenient; it is not ordinary text that should contribute to the sequence meaning.
Suppose the three-token query is padded to five positions:
reset database password [PAD] [PAD]
mask: 1 1 1 0 0A masked mean conceptually computes:
sum of token_vector[i] * mask[i]
-----------------------------------
sum of mask[i]The denominator is 3, not 5.
This matters even if a padding representation looks small or harmless. Contextual token outputs at padded positions depend on the model and masking implementation, so averaging them blindly can change the resulting vector. Use the model’s attention or validity mask to define which positions participate in pooling.
Special tokens require a separate decision. Some embedding models are trained with pooling behavior that includes or excludes particular special or prompt tokens. Follow the checkpoint’s documented encoding procedure rather than applying a generic mask by habit.
Special-token pooling delegates aggregation to one position
Another strategy uses the hidden representation at a designated position, often a classification-style special token such as [CLS] in architectures that define one.
Conceptually:
[CLS] reset database password
|
+---- use this final hidden vector as the sequence embeddingThis is not equivalent to taking an average. The special token participates in transformer attention, so its final representation can incorporate information from other positions. Whether it becomes a good standalone sequence embedding depends on how the model was designed and trained.
That last condition is critical. The mere presence of a [CLS]-like token does not guarantee that its hidden state was optimized for semantic similarity or retrieval. A model trained with a sequence-level objective on that position has a stronger reason to place useful global information there than a model whose training never required that representation to serve as a general-purpose embedding.
Similarly, some causal language-model embedding designs use the final non-padding token rather than a leading special token. The right interpretation comes from the model architecture and training recipe, not from the position alone.
Max pooling keeps the strongest value in each dimension
With max pooling, each output dimension takes the largest value observed across the included token representations.
For the teaching example:
reset -> [ 1.0, 0.2 ]
database -> [ 0.4, 1.4 ]
password -> [ 0.8, 1.0 ]
max pooled -> [ 1.0, 1.4 ]Notice that the two output coordinates can come from different tokens. The pooled vector is not the representation of any one input position.
Max pooling can preserve strong activations that an average would dilute, but it also discards information about how many positions expressed a feature and ignores all non-maximum values in that dimension. As with mean pooling, whether this behavior helps depends on the representations and objective the model learned.
Do not infer token-level explanations directly from a max-pooled dimension unless the model and analysis method justify that interpretation. Hidden dimensions are learned features, not automatically human-readable concepts.
Weighted pooling makes importance explicit
A more general form assigns a weight w_i to each included token vector:
pooled = sum(w_i * h_i) / sum(w_i)For example, suppose a system gives the three token positions weights 1, 2, and 2:
reset -> weight 1
database -> weight 2
password -> weight 2Then:
first dimension = (1*1.0 + 2*0.4 + 2*0.8) / 5 = 0.68
second dimension = (1*0.2 + 2*1.4 + 2*1.0) / 5 = 1.00
weighted embedding = [0.68, 1.00]The arithmetic is easy; choosing defensible weights is the hard part. Weights can come from position rules, learned pooling components, or a model-specific design. Arbitrary hand-written weights may emphasize the wrong evidence.
Weighted pooling is most useful when the weighting rule is part of the model or has been validated for the target task. It should not be treated as an automatic upgrade over an unweighted mean.
Pooling and normalization solve different problems
Embedding pipelines often apply vector normalization after pooling. These operations should not be confused.
Pooling decides which sequence representation to construct. L2 normalization then rescales that resulting vector to unit length:
normalized = pooled / ||pooled||_2For nonzero vectors, cosine similarity between two vectors equals the dot product of their L2-normalized versions. That can make normalized embeddings convenient for systems designed around cosine-style comparison.
But normalization cannot undo a poor pooling choice. If pooling discarded information that mattered to the task, rescaling the resulting vector does not restore it.
The order also matters. Averaging token vectors and then normalizing the pooled result is generally not equivalent to normalizing every token vector first and then averaging. Those are different transformations and can produce different directions. Match the procedure used by the embedding model rather than rearranging steps because they look interchangeable.
Do not change pooling independently of model training
A common mistake is to load an embedding checkpoint, notice that token-level hidden states are available, and substitute a different pooling strategy without evaluating the change.
Suppose a model was trained so that a designated special-token representation is compared using a contrastive loss. Switching to mean pooling at inference changes the representation used for retrieval, even though the transformer weights stay identical. The new vectors may still look numerically reasonable, but they are no longer necessarily the representations the training objective optimized.
The reverse problem also occurs. If a model was trained with masked mean pooling, selecting only its first token because another transformer uses [CLS] pooling imports an assumption from a different model.
Treat these pieces as one contract:
tokenizer + input formatting + encoder + pooling + optional projection + normalizationChanging any part can change embedding geometry and therefore nearest-neighbor rankings.
When using a pretrained embedding model, the safest default is its documented encoding pipeline. Experiment with alternative pooling only when you can evaluate the resulting embeddings on representative data.
Compare pooling strategies on the downstream decision
If you are training or adapting an embedding system and pooling is genuinely a design choice, compare candidates using the task that will consume the vectors.
For semantic retrieval, evaluate whether relevant documents appear near the top of the ranked results. For clustering, use labels or human judgments appropriate to the clustering goal rather than assuming visually compact vectors are sufficient. For a classifier built on frozen embeddings, compare classification performance with the same train/validation split and downstream classifier configuration.
Keep the rest of the pipeline fixed while testing pooling. Otherwise a tokenizer change, normalization change, or new similarity metric can be mistaken for a pooling effect.
Also test realistic sequence lengths. Mean pooling over a five-token query and mean pooling over a long document impose the same equal-weight rule across very different amounts of content. Long inputs can contain several topics, boilerplate, or weakly relevant passages. In such cases, chunking the text and embedding chunks separately may preserve retrieval granularity better than forcing the whole document into one pooled vector.
That is a representation decision, not merely a pooling tweak.
Know when one vector is the wrong abstraction
Pooling is appropriate when the downstream system benefits from one compact vector per input. It reduces storage and makes similarity computation straightforward.
But a single vector is a bottleneck. If a task needs fine-grained matching between parts of a query and parts of a document, a multi-vector or token-level retrieval architecture may retain information that single-vector pooling discards. That approach uses more vectors and usually more comparison work, so the extra fidelity has a storage and latency cost.
Likewise, if you only need exact identifiers or lexical matches, dense embedding pooling may be unnecessary. A lexical retrieval method can be simpler and more predictable for that requirement.
The useful question is not “Which pooling method is best?” It is “What information must survive compression for this model and task?”
Choose pooling as part of the embedding model
Mean pooling is a strong simple mental model: average the valid token representations and obtain one fixed-size vector. Special-token pooling relies on a designated position to aggregate sequence information. Max pooling preserves per-dimension peaks. Weighted pooling makes contribution strength explicit.
None of these rules is universally correct. Their quality depends on how token representations were trained, which positions are included, what post-processing follows, and what the downstream task rewards.
For a pretrained embedding model, preserve its documented pooling and normalization pipeline unless you have evidence to change it. When designing your own model, evaluate pooling as part of the representation architecture rather than as an isolated afterthought. The pooling step is where many token-level states become one decision-making vector, so its assumptions should match the information your application needs to keep.