A language model needs to solve two related problems with vocabulary-sized parameters. At the input, it must turn each token ID into a vector. At the output, it must turn a hidden state into one score for every possible next token. A straightforward architecture gives these two operations separate parameter matrices.
That works, but it can be expensive when the vocabulary and hidden dimension are large. Weight tying is a simple architectural idea: use the same learned matrix for the input token embeddings and the output token projection when their shapes and semantics are compatible.
This article builds a practical mental model for weight tying. You will see exactly which parameters are shared, why the parameter saving can be substantial, how sharing changes optimization, what assumptions it introduces, and when separate matrices are the clearer design.
Start with the two vocabulary-sized matrices
Suppose a small language model has a vocabulary of 10,000 tokens and a hidden dimension of 512.
The input embedding table can be written as a matrix:
E shape: [10000, 512]For token ID 37, the model looks up row E[37] and obtains a 512-dimensional vector. That vector becomes the token representation passed into later layers.
Near the other end of the model, assume the final hidden state for one position is:
h shape: [512]To predict the next token, an untied model can use a separate output matrix:
W_out shape: [10000, 512]
logits = W_out @ hThe result contains 10,000 logits, one per vocabulary item. A softmax can then turn those logits into a probability distribution.
Ignoring an optional output bias, the two matrices contain:
input embeddings: 10000 * 512 = 5,120,000 parameters
output projection: 10000 * 512 = 5,120,000 parametersTogether they require 10.24 million parameters even though both matrices associate vocabulary items with vectors in the same dimensional space.
Weight tying makes one matrix serve both roles
With weight tying, the model keeps E and reuses it at the output:
input vector = E[token_id]
logits = E @ hThe model no longer needs an independent W_out with the same shape. In the example above, the vocabulary-facing matrices therefore use 5.12 million shared parameters instead of 10.24 million separate parameters.
This does not halve the size of the entire model. It removes one vocabulary-by-hidden-dimension matrix. If most parameters live in transformer blocks, the percentage reduction in total model size can be much smaller. The exact saving depends on the vocabulary size, hidden dimension, and the rest of the architecture.
The important mental model is:
untied:
token -> E_in -> model -> W_out -> logits
tied:
token -> E -> model -> E -> logitsThe second use should be understood as a matrix multiplication with the shared parameters, not another embedding lookup.
Why sharing the matrix is meaningful
An embedding row represents a vocabulary item as a learned vector. At the output, a row of the projection matrix acts like a learned direction used to score that same vocabulary item.
When the weights are tied, token 37 uses the same vector in both roles. If its row is e_37, its output logit is the dot product:
logit_37 = e_37 · hA hidden state aligned with e_37 receives a larger score than one pointing in a very different direction, all else equal.
This gives the input and output sides a shared vocabulary geometry. It is a useful constraint, but it is still a constraint. An untied model is free to learn one vector for how a token should be represented as input and another for how that token should be recognized as an output. A tied model requires one parameter vector to support both jobs.
Weight tying therefore trades some parameter freedom for parameter efficiency and shared structure. It should not be interpreted as a guarantee of better accuracy.
The shared weights receive gradients from two paths
Parameter sharing also changes training. In an untied model, the embedding matrix receives gradients through the input path, while the output projection receives gradients through the prediction loss.
With tied weights, both computational paths point to the same parameter matrix. Conceptually, its update reflects gradient contributions from both uses:
shared gradient = contribution from input use
+ contribution from output useAutomatic differentiation systems normally accumulate these contributions when the implementation truly references the same parameter object.
This distinction matters when implementing tying. Copying values once is not the same as sharing weights:
E_out = copy(E_in) # same values now, independent parameters later
E_out = E_in # conceptually shared parameterThe exact syntax depends on the framework, but the architectural requirement is the same: both operations must use one trainable parameter set if the weights are meant to remain tied.
A one-time copy can silently drift apart after the first optimizer step because the two tensors can receive different updates.
Shape compatibility is a real requirement
The simple form of weight tying works cleanly when the input embedding dimension and the representation sent to the output projection have the same size.
For example:
E: [vocabulary_size, 512]
h: [512]
logits = E @ hIf the input embedding dimension is 256 but the final hidden state is 768, the same matrix cannot directly perform both operations:
E: [vocabulary_size, 256]
h: [768]An architecture can introduce an additional projection to reconcile dimensions, but that is no longer the minimal tying pattern. The extra transformation has its own parameters and design implications.
Vocabulary compatibility matters too. If the input and output sides use different vocabularies, a single row cannot automatically mean the same thing on both sides. Encoder-decoder systems, multilingual systems, or specialized output spaces may therefore use different sharing arrangements depending on how their token sets are defined.
Weight tying does not remove the softmax cost
It is easy to confuse parameter reduction with computation reduction.
Reusing the embedding matrix removes a separate set of stored output weights, but a standard full-vocabulary prediction still needs to produce a logit for every vocabulary item. For a hidden state of dimension d and vocabulary size V, the output operation still has the structure of multiplying against a V x d matrix.
So weight tying can reduce parameter storage and the optimizer state associated with the removed parameter matrix, but it does not by itself avoid full-vocabulary scoring. If output computation is the bottleneck, techniques that change how candidates are scored or how the vocabulary is handled address a different problem.
The same distinction applies to inference memory. Tying avoids storing two independent matrices, but runtime memory also includes activations, attention state, caches, temporary buffers, and framework overhead. The end-to-end memory reduction depends on the workload.
Be careful when counting parameters and saving checkpoints
A tied model contains multiple uses of one parameter. Tooling does not always present shared parameters in the most intuitive way.
When validating an implementation, check the properties that matter rather than relying only on layer names:
1. Do input lookup and output scoring reference the same trainable storage?
2. Does one optimizer update keep them tied?
3. Does the parameter count reflect one shared matrix rather than two independent ones?
4. After saving and loading, is the sharing relationship preserved by the model implementation?Serialization formats and frameworks differ in how they represent aliases or reconstruct shared parameters. The architectural guarantee should come from the model definition and its documented loading behavior, not from assuming that two checkpoint entries with similar values must be tied.
This is especially important when converting models between libraries. A converter that materializes two independent tensors may preserve numerical values at conversion time while changing future fine-tuning behavior.
Understand the trade-offs before using it
Weight tying is attractive when the input and output vocabularies correspond directly and the relevant dimensions match. It can remove a large matrix with little conceptual complexity, which is particularly valuable when the vocabulary itself contributes noticeably to parameter count.
The main trade-off is reduced freedom. Separate matrices can specialize: the input table can focus on useful representations for processing context, while the output projection can independently learn directions useful for next-token discrimination. Tying says those roles should share a parameterization.
Whether that constraint helps, hurts, or barely changes task quality depends on the architecture, training setup, and data. Treat it as a design choice to evaluate rather than an automatic improvement.
There are also cases where tying is simply awkward:
- input and output vocabularies differ;
- embedding and output dimensions differ without an intentional bridge;
- an architecture deliberately uses different representations for reading and predicting tokens;
- compatibility with an existing checkpoint requires untied parameters.
In those situations, keeping separate matrices can be simpler and more faithful to the intended model.
A practical way to reason about the choice
When reviewing a language-model architecture, start by locating every large vocabulary-sized matrix. Then ask whether the input embedding rows and output class rows refer to the same token set and have compatible dimensions.
If they do, calculate the actual saving rather than describing tying as vaguely “smaller.” For vocabulary size V and hidden dimension d, removing one independent matrix saves approximately:
V * d trainable parametersThe byte saving for model weights depends on the stored data type. Training can save additional memory because optimizers may keep state for each trainable parameter, but the amount is optimizer- and implementation-dependent.
Finally, verify the implementation as parameter sharing, not value copying, and measure task quality under the same evaluation setup. Parameter efficiency is useful only if the resulting model still meets the product’s quality requirements.
Conclusion
Weight tying is a compact example of an important neural-network design principle: two operations that use compatible semantic structure do not necessarily need independent parameters.
In a language model, the input embedding table maps vocabulary items into vectors, while the output projection scores vocabulary items from a hidden state. Reusing one matrix for both roles can remove V * d parameters and give the two sides a shared vocabulary geometry. In return, the architecture gives up the freedom to learn independent input and output vectors.
Use weight tying when the vocabularies and dimensions line up and the parameter saving is valuable. Keep the matrices separate when their roles, shapes, or token spaces genuinely need to differ. Most importantly, verify that the implementation shares one parameter rather than merely starting two parameters with the same values.