A hyperparameter sweep often leaves you with several fine-tuned models that are individually useful. The usual workflow keeps the checkpoint with the best validation score and discards the rest. An ensemble can use several checkpoints, but then every request may require multiple model evaluations, increasing inference cost and operational complexity.

A model soup offers a third option: average the parameters of compatible fine-tuned models and deploy the resulting parameter set as one model. The technique is simple, but its simplicity can be misleading. Parameter averaging is meaningful only when the checkpoints are sufficiently compatible, and the averaged model still needs independent evaluation.

This article builds a practical mental model for model soups. You will learn what is averaged, why shared initialization matters, how uniform and greedy soups differ, how this technique differs from stochastic weight averaging and prediction ensembles, and when selecting one checkpoint is the safer choice.

Start with parameter averaging

Suppose the same pretrained classifier is fine-tuned three times with different learning rates. To keep the example small, imagine that the model has only two parameters after fine-tuning:

model A: [2.0, 4.0]
model B: [2.2, 3.8]
model C: [1.9, 4.1]

A uniform model soup takes the arithmetic mean of corresponding parameters:

soup = ([2.0, 4.0] + [2.2, 3.8] + [1.9, 4.1]) / 3
     = [2.0333..., 3.9667...]

For a real neural network, the operation is applied element by element to every compatible parameter tensor. If the checkpoints have parameter vectors w_1 through w_n, the uniform soup is:

w_soup = (w_1 + w_2 + ... + w_n) / n

The result is one parameter vector. At inference time, you run that model once. You are not averaging three predictions for every input.

That distinction explains the main deployment attraction: once the soup has been created, its parameter count and ordinary forward-pass structure are the same as those of the constituent architecture. Creating the soup requires storing or reading multiple checkpoints, but serving it does not inherently require keeping those source checkpoints active.

Why averaging weights can work at all

Neural-network parameters are not coordinates with a universal meaning. Two independently trained networks can implement similar functions while storing those functions in very different parameter arrangements. For example, hidden units can be permuted while leaving the network’s overall function unchanged. A naive element-wise average of such unrelated solutions can therefore produce a poor model.

Model soups target a more favorable setting: multiple models are fine-tuned from the same pretrained initialization, usually on the same downstream task. Their training runs may differ in hyperparameters such as learning rate, weight decay, augmentation, or random seed, but they begin from the same parameter organization.

The original model-soups work found that fine-tuned models from a pretrained model can often remain in a region of parameter space where averaging preserves useful behavior. This is an empirical property to test, not a guarantee that all fine-tuned checkpoints can be mixed safely.

A useful mental model is:

shared pretrained starting point
          |
          +--> fine-tuning run A --+
          +--> fine-tuning run B --+--> average weights --> validate
          +--> fine-tuning run C --+

The shared starting point makes parameter correspondence plausible. Validation tells you whether that plausibility held for the runs you actually produced.

Compatibility is stricter than matching file shapes

Two checkpoints can have identically shaped tensors and still be bad candidates for averaging. Before building a soup, check several forms of compatibility.

First, the architecture and parameter names must correspond. If one model adds a different classification head, changes vocabulary size, or inserts adapters that another model does not have, direct full-parameter averaging is not the same operation described above.

Second, the checkpoints should represent the same task semantics. If class index 0 means billing in one classifier and technical in another, averaging their output heads mixes incompatible meanings even if both heads have the same shape.

Third, preprocessing and tokenization assumptions should agree. Parameter averaging does not reconcile different input representations.

Most importantly, shared initialization is a strong practical condition. Matching architecture alone does not establish that corresponding parameters occupy a compatible region of weight space.

Treat checkpoint compatibility as a semantic requirement, not merely a tensor-shape check.

Build a uniform soup first

The simplest useful procedure is a uniform soup. Given a set of candidate checkpoints from the same fine-tuning experiment family:

  1. evaluate each candidate on the same validation set;
  2. choose the candidates that meet your basic quality requirements;
  3. average their corresponding parameters with equal weights;
  4. load the averaged parameters into the same architecture;
  5. evaluate the resulting model as a new model.

In framework-neutral pseudocode:

assert all_checkpoints_are_compatible(checkpoints)

soup = zeros_like(checkpoints[0])
for checkpoint in checkpoints:
    soup += checkpoint / len(checkpoints)

score = evaluate(soup, validation_set)

This example is deliberately simplified. Production code also needs to handle parameter dtypes, non-parameter state, distributed checkpoints, tied parameters, and framework-specific serialization correctly. Do not assume that blindly averaging every numeric value in a checkpoint file is valid.

The important point is the experiment structure: averaging creates a new candidate. It does not inherit the validation score of its ingredients.

Use a greedy soup when some candidates hurt

Uniform averaging treats every selected model equally. That can be wasteful when a hyperparameter sweep contains a few weak or incompatible runs.

A greedy soup uses validation performance to decide which candidates to include. One practical pattern, matching the model-soups idea, is:

1. sort candidate models by validation score
2. initialize the soup with the strongest candidate
3. consider each remaining candidate in order
4. temporarily average it with the current soup
5. keep it only if the validation metric improves

Suppose four checkpoints score as follows:

A: 91.0
B: 90.8
C: 90.4
D: 86.2

Starting from A, adding B might produce a soup scoring 91.3. Adding C might raise it to 91.4. Adding D might lower it to 90.7, so D is rejected.

The numbers are illustrative rather than a claim about typical gains. What matters is the decision rule: validation data determines whether an additional checkpoint belongs in the soup.

There is also a statistical caution. If you try many soup combinations and repeatedly choose whichever scores highest on one small validation set, you can overfit the selection process to that set. Keep a final test set or other untouched evaluation data for the final comparison.

Do not confuse model soups with ensembles

A prediction ensemble keeps multiple models and combines their outputs. For three classifiers, a simplified ensemble might compute:

prediction = mean([
    model_A(x),
    model_B(x),
    model_C(x),
])

A model soup instead averages parameters once:

soup_weights = mean([weights_A, weights_B, weights_C])
prediction = soup_model(x)

These operations are generally not mathematically equivalent because neural networks are nonlinear functions of their parameters. Averaging three parameter vectors does not, in general, produce a model whose output equals the average of the three original outputs.

An ensemble can therefore retain diversity that parameter averaging loses. Its cost is that it normally requires multiple forward passes and multiple model states at inference. A soup trades that diversity for a single deployable model. Whether the trade is favorable is an evaluation question.

Model soups are also different from stochastic weight averaging

Model soups and stochastic weight averaging (SWA) both average parameters, so they are easy to confuse.

SWA usually collects multiple points from the trajectory of one training run, typically late in training under a suitable learning-rate schedule, and averages those points. The averaging procedure is part of the training strategy.

A model soup instead combines separate fine-tuning runs, often produced by a hyperparameter sweep from the same pretrained initialization. It is primarily a post-training way to reuse multiple completed candidates.

The distinction affects when each technique is useful:

SWA:
one training trajectory -> several late checkpoints -> average

model soup:
shared pretrained model -> several fine-tuning runs -> average

Neither label makes arbitrary checkpoint averaging safe. In both cases, the useful result depends on the geometry and compatibility of the parameter solutions being combined.

Evaluate more than the metric used to build the soup

A soup that improves one validation metric can still create regressions elsewhere. Evaluate it with the same discipline as any other new model version.

For a classifier, that can include per-class precision and recall, calibration, slice-level performance, and behavior under distribution shifts that matter to the application. For a language model or embedding model, use task-specific evaluations that reflect the actual deployment objective rather than assuming that parameter averaging preserves every capability.

Also compare operational properties. A soup does not inherently add inference passes, but the checkpoint may still differ in numerical values that affect downstream quantization, calibration, or other post-training steps. If deployment includes quantization or compilation, validate the final deployed artifact rather than only the full-precision soup.

A sound comparison often includes at least:

best single checkpoint
uniform soup
selected or greedy soup
optional prediction ensemble as an upper-cost reference

This separates three questions: whether averaging helps, whether candidate selection helps, and whether a more expensive ensemble buys enough additional quality to justify its serving cost.

Common failure modes

Averaging independently initialized models

Two models can have the same architecture and similar accuracy while representing their internal features with different parameter arrangements. Element-wise averaging can then destroy useful structure. Same architecture is not enough; shared initialization and empirical compatibility matter.

Mixing incompatible heads or label mappings

If downstream heads represent different label orders or tasks, their corresponding coordinates do not mean the same thing. The resulting soup can be syntactically loadable and semantically wrong.

Assuming every sweep run deserves equal weight

A failed or badly tuned run can pull a uniform average toward a worse region. Inspect individual candidates and compare a uniform soup with a selection procedure rather than treating more ingredients as automatically better.

Selecting and reporting on the same data repeatedly

Greedy inclusion decisions consume information from the validation set. Reporting the final score on that same repeatedly consulted set can make the result look more reliable than it is. Reserve independent evaluation data for the final claim when the stakes justify it.

Averaging checkpoint state without understanding it

Optimizer moments, learning-rate scheduler state, counters, and other training metadata are not model parameters that should automatically be averaged. If the goal is an inference model, average only state whose semantics you understand and reconstruct any required non-parameter state appropriately.

When a model soup is worth trying

Model soups are especially attractive when you already paid for several compatible fine-tuning runs and want to extract more value from them without deploying an inference-time ensemble. In that setting, producing and evaluating an averaged checkpoint can be inexpensive compared with launching another training sweep.

They are also useful as an experiment when several strong fine-tuned models come from the same pretrained checkpoint and differ only in reasonable training choices. The technique gives you a way to test whether those nearby solutions can be combined into one stronger deployment candidate.

A simpler approach is better when you have only one strong checkpoint, when serving an ensemble is affordable and its extra quality is important, or when the available models were trained independently and have no reason to be compatible in parameter space.

Do not use model soups as a substitute for resolving conflicting task definitions. If two models encode different label semantics, safety policies, vocabularies, or preprocessing contracts, averaging their weights does not reconcile those differences.

Conclusion

A model soup is best understood as post-training parameter averaging among compatible fine-tuned models. Its appeal is practical: multiple completed training runs can sometimes be compressed into one deployment model without requiring multiple inference passes.

The key constraint is compatibility. Start from the same pretrained model, keep architecture and task semantics aligned, build a uniform average as the simplest baseline, and use validation-guided selection only when it earns its complexity. Then evaluate the averaged checkpoint as a new model on untouched data and on the operational properties that matter in deployment.

Weight averaging is cheap. Trusting the result without testing it is not.

Artificial Intelligence Machine Learning Neural Networks Fine-Tuning