Model Merging

Combine Fine-Tuned Models with Weight Averaging

Fine-tuning the same model for different datasets or objectives can leave a team with several useful checkpoints. Serving all of them as an ensemble may improve robustness, but it also multiplies inference work. Choosing only one checkpoint avoids that cost but discards what the others learned. Weight averaging offers a third option: combine compatible checkpoints by averaging their parameters, then serve the result as one model. The arithmetic is simple. The important question is whether the checkpoints occupy a compatible region of parameter space so that interpolation preserves useful behavior rather than destroying it.