A language model can contain two large matrices indexed by the same vocabulary: one maps token IDs into embedding vectors, while another maps hidden states into vocabulary logits. Weight tying makes those roles share parameters instead of maintaining two independent matrices.

The change is compact in code, but it affects parameter counting, gradient flow, dimensional constraints, checkpoint handling, and any component that assumes the input and output weights are separate.

The same matrix can serve two directions

Let the vocabulary size be V and the embedding width be d. An input embedding matrix can be written as:

E in R^(V x d)

For token ID i, the model selects row E[i] as its embedding.

At the output, a hidden vector h with width d needs one score per vocabulary item. An untied model can use a separate matrix W:

logits = W h + b
W in R^(V x d)

With weight tying, W and E refer to the same parameter values:

logits = E h + b

The input path still performs row lookup, while the output path uses the matrix as a projection over the full vocabulary. Sharing parameters does not make those computations identical; it makes them consume the same matrix.

Tying removes a vocabulary-sized parameter matrix

If both untied matrices have shape V x d, replacing them with one shared matrix removes V * d independently stored parameters. The output bias, if present, remains separate.

That reduction can be substantial when the vocabulary is large, but it should not be confused with halving the whole model. Transformer blocks, normalization parameters, biases, and other components are unaffected.

Memory accounting also depends on training state. An optimizer can keep auxiliary tensors for each trainable parameter. When two conceptual roles truly reference one parameter, optimizer state should correspond to that shared parameter rather than to two copied matrices.

Copying values once is not weight tying. Two tensors initialized with identical numbers can diverge after the first update if they remain independent parameters.

Gradients from both roles meet at one parameter

A tied matrix participates in two parts of the computation graph. It receives gradient contributions through token embedding lookups and through the output projection.

Conceptually, if the loss depends on the shared matrix through both paths, its total gradient contains both contributions:

dL/dE = input_path_contribution + output_path_contribution

Automatic differentiation can accumulate these contributions when both operations reference the same parameter object. A custom implementation that duplicates storage and manually synchronizes values has different semantics unless it also handles gradients and optimizer state consistently.

This shared gradient path is one reason parameter identity matters more than equal tensor contents.

Dimensions can block direct tying

Direct tying requires compatible shapes. If token embeddings have width d_e while the final hidden representation has width d_h, the same V x d_e matrix cannot directly multiply a d_h vector when d_e != d_h.

A model can insert an additional projection between hidden states and the tied vocabulary matrix:

z = P h
logits = E z + b

Here P maps the hidden width to the embedding width. This preserves a shared vocabulary matrix but adds another transformation, so it is architecturally different from direct tying.

Vocabulary alignment is another constraint. Input and output roles must agree on which row represents each token. Sharing storage across mismatched token-index mappings would attach an output score to the wrong embedding row.

Serialization must preserve sharing semantics

A checkpoint format may store tensor values without preserving object aliasing automatically. Loading two copies of the same numeric matrix into two independent parameters does not recreate tied weights.

Frameworks and model libraries differ in how shared parameters are declared, serialized, and reconstructed. A reliable load path should verify parameter identity or the framework’s documented sharing mechanism after restoration, not merely compare values immediately after loading.

The same concern applies to model conversion and export. A target runtime may represent shared initializers differently or may materialize duplicates for execution. That can preserve numerical inference while changing memory usage.

Resizing the vocabulary touches both roles

Adding or removing vocabulary entries changes the row count of the embedding matrix. In a tied model, that matrix also defines the output projection rows, so vocabulary resizing affects both ends at once.

A resize operation must keep token IDs, embedding rows, output logits, and any output bias aligned. Extending only the input lookup while leaving an older output projection would break the tying relation. Replacing the shared parameter with a new matrix also requires the optimizer to reference the replacement if training continues.

This coupling is useful because one vocabulary definition governs both roles, but it makes ad hoc resizing harder to isolate.

Sharing is an architectural constraint, not a storage trick

Weight tying is sometimes described mainly as parameter reduction. That description misses the stronger property: the model is constrained to use one vocabulary representation matrix for both token lookup and output scoring.

That constraint affects optimization because both uses update the same values. It affects implementation because dimensions and token indices must align. It affects tooling because serialization, resizing, quantization, and conversion code must preserve or intentionally relax the sharing relationship.

For developers, the key distinction is between shared parameter identity and duplicated values. The former defines tied weights; the latter only creates two matrices that happen to start at the same point.

Artificial Intelligence Language Models Neural Networks Embeddings