Scaling a neural network creates an expensive tuning problem. A learning rate that works for a small prototype may behave differently after hidden dimensions become much wider. If every model size needs a fresh hyperparameter sweep, experimenting on small models saves less compute than it first appears.

Maximal Update Parametrization, usually written MuP or μP, addresses this problem by changing how parameter initialization and learning rates scale with model width. The goal is not to make a wider model identical to a narrow one. It is to make important training dynamics behave consistently enough that hyperparameters tuned on a smaller proxy can often transfer to a wider target.

This article explains the mental model behind MuP, what widthwise hyperparameter transfer actually promises, why ordinary parametrization can break transfer, and how to design a practical proxy-to-target experiment without treating MuP as a universal scaling rule.

Width changes more than the parameter count

Consider a hidden representation with width n. A linear layer maps it through a weight matrix:

h_next = W h

If n increases, more terms contribute to each output and more parameters participate in learning. Initialization schemes normally scale weights so activations do not explode merely because the layer became wider. But stable activations at initialization are only part of the problem.

Training also changes those weights. The same optimizer learning rate can produce representation updates with a different scale when width changes. A parametrization that looks reasonable at one width can therefore become too conservative or too aggressive at another.

This is why “train a tiny model, then copy its learning rate to a huge model” is not automatically a controlled experiment. The two models may use the same optimizer setting while operating in different width-dependent training regimes.

MuP treats that width dependence as something to design explicitly.

Think in terms of comparable updates

A useful mental model is to separate three quantities:

parameter scale at initialization
        +
optimizer update scale
        +
width of the layer
        |
        v
scale of changes seen by network features

MuP chooses width-dependent parameter and update scalings so that widening the network approaches a well-behaved feature-learning limit. The word maximal refers to keeping useful feature updates as large as possible without making them diverge in the infinite-width analysis behind the parametrization.

You do not need to take an infinite-width limit when training a practical model. The limit supplies the scaling rules. The engineering benefit is that finite models at different widths can then exhibit more compatible hyperparameter behavior.

This distinction matters: MuP is not simply a new weight initializer, and it is not simply a learning-rate schedule. It is a coordinated parametrization of the network and optimizer.

A proxy-model experiment shows the intended workflow

Suppose you plan to train a transformer with hidden width 4096, but trying ten learning rates at that size is too expensive. You construct a smaller model with the same relevant architecture but hidden width 512.

Without a transfer-aware parametrization, the workflow is uncertain:

width 512 -> tune learning rate -> 3e-4 works well
width 4096 -> copy 3e-4 -> behavior may shift with width

With MuP, the intended workflow is:

define one width family under MuP
        |
        +-> proxy width 512 -> tune transferable hyperparameters
        |
        `-> target width 4096 -> apply corresponding MuP scaling
                                + transfer tuned values

The transferred value is a base hyperparameter interpreted through the parametrization. Some parameters can require width-dependent optimizer scaling even though the user-facing base learning rate stays fixed.

The important result is therefore not “all layers use exactly the same numeric step size at every width.” It is “the parametrization defines how each parameter group should scale so that the base hyperparameter can remain transferable.”

Not every parameter scales the same way

Neural networks contain parameters with different structural roles. A matrix whose input dimension grows with model width is not equivalent to a bias vector or a parameter whose dimensions remain fixed.

MuP classifies parameters according to how their dimensions change across the model family and applies scaling rules appropriate to those roles. In practical implementations, this often means that initialization multipliers and optimizer learning-rate multipliers differ across parameter groups.

That is why manually adding one global factor such as

learning_rate / sqrt(width)

is not a general implementation of MuP. A single heuristic ignores which dimensions are widening and what role each tensor plays in the network.

For production use, rely on a tested MuP implementation or derive the parametrization carefully for the architecture. Small mistakes can leave the model trainable while silently destroying the transfer property you wanted to measure.

Define the model family before tuning

Hyperparameter transfer only makes sense relative to a family of related models. Before running a proxy sweep, decide what changes between proxy and target.

For widthwise transfer, a clean family might keep these properties fixed:

number of transformer blocks
vocabulary
training objective
data pipeline
tokenization
optimizer family
learning-rate schedule shape

while changing hidden width and the dimensions that depend on it.

This does not mean every one of those properties must always remain fixed in research. It means that changing several axes at once makes the transfer experiment harder to interpret. If the proxy has different depth, data mixture, sequence length, optimizer, and width, a failed transfer does not tell you which change caused the mismatch.

MuP’s established use is widthwise hyperparameter transfer. Do not assume that the same rules automatically provide transfer across depth, architecture families, or unrelated training objectives.

Separate transferable and non-transferable choices

The phrase “hyperparameter transfer” can sound broader than it is. A useful workflow labels each choice before the sweep.

Potentially transferable quantities in a MuP setup can include base learning-rate-like hyperparameters and other values for which the parametrization was designed and empirically checked. But model-independent configuration values, architecture choices, and data-dependent settings have different semantics.

For example, changing batch size changes gradient statistics and the number of optimizer steps for a fixed amount of data. Changing sequence length changes the computation and examples seen in each step. MuP does not make those changes irrelevant merely because width transfer works.

Treat the proxy as an experiment on a controlled model family, not as a cheap oracle for every target-model decision.

Verify transfer at more than one width

Jumping directly from one tiny proxy to one expensive target makes failures difficult to diagnose. A stronger procedure uses at least one intermediate width when budget permits.

Suppose a learning-rate sweep on width 256 prefers 3e-4. Test a small set around that value at width 512 or 1024:

width 256:   1e-4   3e-4   1e-3
                       ^ best validation result

width 512:   1e-4   3e-4   1e-3
                       ^ still competitive

These numbers are illustrative, not recommended defaults. The point is experimental design. If the preferred region moves dramatically at the intermediate width, investigate the parametrization before paying for the largest run.

A transfer check should compare learning curves as well as one final metric. Look for instability, different convergence speed, or a systematic shift in the hyperparameter optimum. A single final score can hide a training dynamic that is already diverging across widths.

Keep compute comparisons fair

A wider model usually performs more computation per training step. It may also require a different device layout or achieve different hardware utilization. MuP does not remove those systems effects.

When comparing widths, state what is held constant:

same number of optimizer steps?
same number of training tokens?
same batch size?
same wall-clock budget?
same learning-rate schedule in step units?

These choices answer different questions. If a schedule decays after 10,000 steps in every run but batch size changes with width, the models may see different amounts of data before the decay. That can look like a failure of hyperparameter transfer even though the training protocols were not equivalent.

The parametrization controls width-related mathematical scaling. Experimental bookkeeping still controls whether the comparison is meaningful.

Know what MuP does not guarantee

MuP can reduce a particular source of hyperparameter drift, but several limits remain important.

First, finite models are not the mathematical infinite-width limit. Transfer is an empirical property to validate, especially when the proxy is extremely narrow.

Second, a correctly parametrized target can still train poorly because of bad data, an unsuitable optimizer, numerical instability, insufficient capacity, or an inappropriate objective. MuP does not diagnose those problems.

Third, changing architecture can invalidate assumptions used to define the model family. Adding new branches, changing which dimensions widen, tying or untying parameters, or altering normalization can require revisiting the parametrization.

Fourth, successful width transfer does not imply successful depth transfer. Scaling the number of layers changes signal propagation and residual dynamics in ways that are not captured by ordinary widthwise MuP rules.

Finally, hyperparameter optima can be broad rather than exact. If several learning rates perform similarly on the proxy, reporting one as a precisely transferable optimum overstates what the experiment established.

When MuP is worth the complexity

MuP is most attractive when the target model is expensive enough that direct hyperparameter sweeps are a major part of the training budget, and when you can construct cheaper models that differ mainly in width.

It is also useful when studying scaling behavior because it reduces one confound: changing width no longer necessarily means accepting arbitrary changes in update scale from the parametrization itself.

A simpler approach can be better when the target is cheap to tune directly, when only one model size will ever be trained, or when proxy and target differ across many architectural dimensions. In those cases, the engineering effort needed to define and validate a MuP family may exceed the tuning cost it saves.

Do not adopt MuP merely because a model is large. Adopt it when widthwise transfer solves a concrete experimental or compute-budget problem.

Build a transfer test before the expensive run

A practical validation sequence is:

1. Choose the width dimension you intend to scale.
2. Define proxy and target models as one architecture family.
3. Apply MuP consistently to every width-dependent parameter group.
4. Tune candidate base hyperparameters on a cheap proxy.
5. Test the preferred region at an intermediate width.
6. Compare learning curves and validation metrics across widths.
7. Launch the target only after the transfer behavior is credible.

Record both the base hyperparameters and the derived per-parameter scaling. Saving only a global learning rate is not enough to reproduce a MuP run if different parameter groups receive different effective rates.

Also record the proxy width, target width, initialization policy, optimizer, schedule, and any implementation version that determines parameter classification. Reproducibility depends on the complete parametrization, not just the label “MuP.”

Conclusion

Maximal Update Parametrization is a way to make neural-network width scaling more controlled. It coordinates initialization and optimizer scaling so that wider models can preserve comparable feature-learning dynamics, which can make hyperparameters tuned on smaller proxies useful at larger widths.

The practical value comes from disciplined use: define one width family, implement the parameter-specific scaling correctly, tune on a cheap proxy, and verify transfer at an intermediate size before committing to the expensive target. MuP does not make all model changes transferable, but when width is the main scaling axis, it can turn small-model experiments into more informative guides for large-model training.

Artificial Intelligence Machine Learning Neural Networks Model Training