Language models need to turn token IDs into vectors before processing them and turn hidden vectors back into vocabulary scores before predicting the next token. A straightforward design gives those two operations separate parameter matrices. When the vocabulary and hidden dimension are large, each matrix can contain many parameters.
Weight tying removes that duplication by reusing one parameter matrix for both roles. The input side reads rows from the matrix as token embeddings; the output side uses the same learned vectors to score candidate tokens, usually through the matrix transpose.
The idea is small, but it teaches a useful model-design principle: two layers can share parameters when their shapes and semantics are compatible. This article builds the idea from a tiny example, shows exactly what is shared, explains the parameter savings, and covers cases where tying is inappropriate or needs extra care.
Start with two vocabulary-sized matrices
Assume a language model has a vocabulary of 10,000 tokens and a hidden size of 512.
The input embedding table has shape:
10,000 x 512It therefore contains:
10,000 * 512 = 5,120,000 parametersFor token ID 37, the embedding lookup selects row 37 and passes that 512-dimensional vector into the model.
At the output, suppose the final hidden state is also 512-dimensional. To produce one score, or logit, for every vocabulary token, an untied output projection can use a second matrix with shape:
512 x 10,000That is another 5,120,000 parameters, before any optional output bias.
The two matrices face opposite directions, but they connect the same two spaces: vocabulary items and 512-dimensional model representations. Weight tying exploits that compatibility.
Reuse the embedding matrix for output scores
Let the embedding matrix be E with shape V x D, where:
Vis vocabulary size;Dis embedding and hidden size.
An input token i uses row E[i] as its embedding.
If the final hidden vector h has dimension D, an untied model could learn a separate output matrix W and compute:
logits = h Wwith W shaped D x V.
With tied weights, the model instead computes:
logits = h E^TE^T has shape D x V, so the multiplication still produces V logits. The model can then apply softmax when probabilities are needed.
For one candidate token j, its logit is the dot product:
logit[j] = h dot E[j]This gives the shared row E[j] two jobs. On input, it represents token j for the network. On output, it acts as the vector against which the final hidden state is compared when scoring token j.
Weight tying does not mean the input and output computations are identical. Lookup selects one or more rows; output scoring compares a hidden vector against every vocabulary row. What is shared is the parameter storage and the values learned in those rows.
See the parameter saving directly
Without tying, the two vocabulary-facing matrices contain approximately:
input embedding: V * D
output matrix: D * V
--------------------------------
total: 2 * V * DWith tying, they contain:
shared matrix: V * DFor the 10,000-token, 512-dimensional example, tying removes 5,120,000 independent matrix parameters.
The saving becomes more noticeable as V or D grows. It can reduce model checkpoint size and the memory needed to store parameters. The effect on total model size depends on the rest of the architecture: in a deep transformer, attention and feed-forward layers may still account for most parameters.
Do not translate parameter savings directly into an equal inference-speed improvement. Autoregressive generation still needs to produce vocabulary logits unless the serving system uses another output strategy. Tying avoids a second set of learned weights; it does not eliminate the output projection computation itself.
Understand what training does to shared weights
Parameter sharing also changes optimization. Because E participates in both input embedding and output scoring, gradients from both uses contribute to updates of the same parameter matrix.
Consider the row for a token such as database. During training, that row can receive learning signal because database appeared in the input context. It can also receive signal through the output objective when the model assigns probability across candidate next tokens.
The important point is not that these signals are guaranteed to agree. They are optimized jointly because they update the same parameters. Tying therefore acts as an architectural constraint: the model cannot independently learn one vector for reading a token and an unrelated vector for scoring that token at the output.
That constraint can be useful when the two roles should share structure, but it also removes flexibility. Whether the trade-off helps a particular model is an empirical question rather than a universal guarantee.
Check dimensions before tying
The simplest form of weight tying requires the input embedding dimension to match the dimension of the vector sent to the vocabulary projection.
Suppose the token embedding has dimension 256, but the model’s final hidden state has dimension 768. Then this expression is not directly valid:
h E^T
h: 768
E^T: 256 x VThe inner dimensions do not match.
One design can insert a learned projection that maps the final hidden state from 768 dimensions to 256 before applying E^T:
768-dimensional hidden state
|
v
learned projection
|
v
256-dimensional vector
|
v
E^T -> vocabulary logitsThat makes tying possible, but the projection adds parameters and computation. It is a different architecture from simply setting the hidden and embedding dimensions equal, so its quality and cost should be evaluated rather than assumed.
Do not tie weights just because the shapes match
Shape compatibility is necessary, but semantics matter too.
Different input and output vocabularies
Some sequence-to-sequence systems use different source and target vocabularies. If input token ID 42 means one symbol while output ID 42 means another, sharing rows by numeric ID has no useful interpretation.
Sharing can still make sense when source and target use the same tokenizer and vocabulary, but that is a model-design choice rather than a requirement.
Different roles may need independent capacity
Even with one vocabulary, an architecture may intentionally keep input embeddings and output weights separate. Untied matrices let the model optimize the two representations independently. That costs more parameters but removes the tying constraint.
If you are comparing tied and untied variants, keep the evaluation objective fixed. A lower parameter count is valuable only if model quality, memory, latency, and deployment constraints remain acceptable for the application.
Output bias is separate
Tying the embedding and output matrix does not automatically tie or remove an output bias. A model may still learn a vocabulary-sized bias vector if its architecture defines one. Treat matrix tying and bias design as separate decisions.
Implement sharing as actual parameter sharing
A subtle implementation mistake is to initialize two matrices with equal values but keep them as separate trainable parameters:
input_weights = copy(initial_matrix)
output_weights = copy(initial_matrix)They start equal, but independent gradient updates can make them diverge immediately. That is not weight tying.
Conceptually, both operations must reference the same trainable parameter:
E = trainable_matrix(V, D)
input_embedding(token_id) = E[token_id]
output_logits(h) = h @ transpose(E)Framework-specific APIs differ, so verify parameter identity rather than relying only on matching initial values. A useful implementation test is to confirm that the optimizer sees one shared matrix instead of two independent vocabulary-sized matrices.
Checkpoint loading deserves the same attention. If an untied checkpoint contains separate input and output matrices, converting it to a tied architecture requires an explicit policy for which values become the shared matrix. Silently assuming the matrices are interchangeable can change model behavior.
Measure the trade-off that matters to deployment
Weight tying is attractive when vocabulary-facing parameters are a meaningful part of the model budget. Evaluate at least three effects.
First, count parameters before and after tying. The expected reduction for a simple tied input/output pair is V * D parameters, assuming the untied design used two full matrices of compatible size.
Second, measure task quality using the same validation data and decoding settings. Tying changes the hypothesis space, so parameter reduction alone cannot tell you whether the model remains suitable.
Third, measure the resource you actually care about. Fewer parameters can reduce storage and parameter memory, but end-to-end latency may be dominated by other layers, memory movement, batching, or token generation. Benchmarking the deployed workload is more informative than inferring latency from parameter count.
For a small model where the embedding and output matrices dominate the parameter budget, tying can be a substantial simplification. For a large architecture where they are a small fraction of total parameters, the relative saving may be modest.
When weight tying is a good fit
Weight tying is worth considering when the model reads and predicts tokens from the same vocabulary, the relevant representation dimensions are compatible, and reducing parameters or checkpoint size is useful. It is especially natural for language-model architectures in which input embeddings and next-token output scores operate over the same token set.
Keep separate weights when the input and output token spaces have different meanings, when the dimensions make tying awkward without extra projections, or when experiments show that independent input and output representations provide a worthwhile quality advantage.
The broader lesson is reusable beyond language models. Parameter sharing is not merely compression: it encodes an assumption that two parts of a model should learn a common representation. Weight tying works cleanly when that assumption matches both the tensor shapes and the problem semantics.