A machine learning model can fail because of patterns specific to its training run: its initialization, sampled batches, training data, architecture, or hyperparameters. Training another model may produce different mistakes. Ensembling uses that disagreement by combining predictions from multiple models instead of trusting one model alone.
The idea is simple, but useful ensembles require more than averaging everything available. Models that make nearly identical errors provide little complementary information, while diverse models can improve predictions at the cost of additional training, memory, and inference work.
This article develops a practical mental model for ensembling, shows how to combine classification and regression predictions correctly, and explains how to decide whether the improvement is worth the operational cost.
Start with three imperfect classifiers
Suppose three binary classifiers estimate whether a transaction is fraudulent. For one transaction they output probabilities:
model A: 0.80
model B: 0.65
model C: 0.70A simple soft-voting ensemble averages the probabilities:
(0.80 + 0.65 + 0.70) / 3 = 0.7167If the application uses a threshold of 0.60, the ensemble predicts fraud.
This is different from hard voting, where each model first converts its score into a class and the ensemble chooses the class receiving the most votes. Soft voting preserves more information because a prediction of 0.51 is treated differently from 0.99.
For regression, the simplest equivalent is also an average. If three models predict delivery times of 28, 31, and 30 minutes, their mean prediction is:
(28 + 31 + 30) / 3 = 29.67 minutesThese examples show the mechanism. They do not yet explain why averaging can help.
Ensembling helps when errors are not perfectly aligned
Imagine two regression models whose predictions can be written as:
prediction A = true value + error A
prediction B = true value + error BTheir average is:
average prediction = true value + (error A + error B) / 2If one model tends to err high on examples where the other errs low, some error cancels. If both make the same error on the same examples, averaging does not remove it.
This leads to the central mental model:
An ensemble gains value from useful differences between its members, not from member count by itself.
For example, five copies of the same deterministic model trained on identical data with identical initialization will produce the same prediction. Their average is still that same prediction. There is no ensemble benefit.
By contrast, models trained with different random seeds, data samples, architectures, feature sets, or training procedures can learn somewhat different decision functions. Whether those differences are useful must be measured on held-out data.
Accuracy and diversity must be considered together
Diversity alone is not enough. A random classifier is very different from a strong classifier, but adding it to an ensemble may make predictions worse.
A useful member should therefore satisfy two properties:
- it performs reasonably well on its own;
- its errors are not completely redundant with the errors of the other members.
Consider two candidate models with the same validation accuracy. If model B fails on almost exactly the same examples as model A, combining them may change little. If model C fails on a different subset, A and C may form the stronger pair.
You can inspect this directly rather than relying only on aggregate metrics. For classification, record whether each model is correct for every validation example and examine how often pairs fail together. For regression, inspect correlations between residuals:
residual = prediction - targetHigh residual correlation means the models tend to err in the same direction on the same examples. Lower correlation can make averaging more useful, provided each model is individually competent.
Do not optimize for low correlation in isolation. A weak model with unusual errors can have low correlation and still damage the ensemble.
Average probabilities, not logits, unless you intend different semantics
Neural classifiers commonly produce logits, which are unnormalized scores before the sigmoid or softmax transformation.
Suppose two binary classifiers output logits:
model A logit: 2.0
model B logit: -1.0There are at least two possible combinations:
average logits -> apply sigmoid
average sigmoid probabilitiesThese operations are not generally equivalent because sigmoid is nonlinear:
sigmoid((2 + -1) / 2) != (sigmoid(2) + sigmoid(-1)) / 2Numerically:
sigmoid(0.5) ~= 0.622
(sigmoid(2.0) + sigmoid(-1.0)) / 2 ~= 0.575Neither rule is universally correct for every ensemble design, but they represent different models. If your intended interpretation is “each member supplies a probability and each member has equal weight,” average probabilities.
Averaging logits instead combines evidence on the log-odds scale for binary classification. Use it only when that behavior is deliberate and validated.
The same principle applies to multiclass models: decide whether you are combining class probabilities, logits, votes, or another score, and document that choice.
Probability calibration affects soft voting
Soft voting treats predicted probabilities as quantities worth averaging. If one member is severely overconfident, it can exert more practical influence on the ensemble even when every model receives the same arithmetic weight.
Consider two models:
model A: [0.99, 0.01]
model B: [0.40, 0.60]Their mean is:
[0.695, 0.305]Model A dominates the result because its distribution is much sharper.
That may be appropriate if A’s confidence is meaningful. It may be harmful if A is simply overconfident. Evaluate both member calibration and ensemble calibration when downstream decisions depend on probabilities rather than only class ranking.
Calibration is not a prerequisite for every ensemble, but it becomes important when probability values drive thresholds, expected-cost decisions, or risk estimates.
Weighted ensembles can help, but add another fitted component
Equal averaging is a strong baseline because it has no extra parameters. Sometimes members have consistently different quality, and a weighted combination performs better:
ensemble = 0.5 * model A
+ 0.3 * model B
+ 0.2 * model CFor probability vectors, non-negative weights that sum to one keep the result a valid probability distribution when each member output is valid.
The danger is choosing weights on the test set or repeatedly tuning them against the same validation set until they overfit. Ensemble weights are model parameters in an operational sense: they were selected using data and therefore need honest evaluation.
A clean workflow is:
training data -> fit member models
validation data -> choose members and ensemble rule
final test data -> estimate final performance onceIf data is scarce, cross-validation or out-of-fold predictions can support more careful ensemble construction without using the final test set for tuning.
Stacking learns how to combine models
Averaging assumes the same combination rule everywhere. Stacking trains another model, often called a meta-model, to combine member predictions.
For example, three classifiers might produce:
[p_A, p_B, p_C]A logistic regression model can use those values as features and learn how they relate to the target.
The important implementation detail is how the meta-model’s training features are generated. If each base model predicts examples that it was trained on, those predictions can be unrealistically good. The meta-model then learns from leaked information.
A safer approach uses out-of-fold predictions:
split training data into folds
for each fold:
train base models on the other folds
predict the held-out fold
combine all held-out predictions
train the meta-model on those predictionsAfter the combination rule is established, base models can be retrained according to the deployment plan.
Stacking can capture useful relationships that simple averaging misses, but it increases pipeline complexity and creates another place to overfit. Start with averaging and require measured evidence before adding a learned combiner.
Deep ensembles use independent training runs
For neural networks, a common approach is to train the same architecture multiple times with different random initialization and training randomness, then average their predictions.
Although the architecture and dataset are the same, non-convex optimization can lead different runs to different parameter values and somewhat different predictions. The resulting deep ensemble can improve predictive performance and can provide a disagreement signal useful for uncertainty analysis.
For a scalar prediction from M members, compute:
mean = sum(prediction_i) / Mand inspect spread around that mean. Large disagreement can indicate that the members do not agree about the input.
However, disagreement is not a guaranteed measure of real-world uncertainty. All members can confidently make the same wrong prediction, especially when they share the same training data and modeling assumptions. Treat ensemble spread as a signal that needs validation, not as a proof that uncertainty has been quantified correctly.
Measure marginal value as the ensemble grows
Adding models usually increases cost roughly with the amount of extra model execution, but quality gains often show diminishing returns.
Suppose validation results look like this:
1 model: F1 = 0.842
2 models: F1 = 0.858
3 models: F1 = 0.863
4 models: F1 = 0.864The fourth model adds almost no measured quality in this example. If production inference cost rises by roughly another model execution, that trade-off may be unattractive.
Do not assume this pattern will hold for every task. Instead, build an ensemble-size curve: evaluate the metric that matters after adding each candidate member. Also record latency, memory, and compute cost.
The deployment question is not “How many models can we ensemble?” It is “Where does the next member stop earning its cost?”
Inference cost is often the main downside
If three independent models must run for every request, the system performs substantially more computation than a single-model service. Whether latency triples depends on execution strategy and hardware: models may run partly in parallel, but parallel execution also requires enough memory and compute capacity.
Important costs include:
- storing multiple sets of parameters;
- loading or keeping several models resident;
- executing each model for every prediction;
- combining outputs;
- monitoring and versioning multiple artifacts.
For a batch offline job, these costs may be acceptable. For a latency-sensitive service on constrained hardware, they may dominate the quality improvement.
If an ensemble produces a useful accuracy gain but is too expensive to serve, knowledge distillation is one possible next step: train a smaller student to approximate the ensemble’s behavior. Distillation can recover some benefits, but the student is a new model and must be evaluated independently rather than assumed to match the ensemble.
Common mistakes
Ensembling duplicate models
More members do not guarantee more useful diversity. Measure pairwise error behavior and marginal ensemble improvement.
Selecting members on the test set
Choosing the best ensemble after examining test performance makes the test set part of model development. Keep final evaluation data separate from member and weight selection.
Combining incompatible outputs
Probabilities from models with different class orders cannot be averaged safely until labels are aligned. The same applies when preprocessing, target definitions, or output semantics differ.
Ignoring calibration
An ensemble can improve accuracy while still producing poorly calibrated probabilities. Evaluate the property the application actually consumes.
Comparing against a weak single-model baseline
An ensemble’s extra cost should be justified against a well-tuned single model, not merely the first baseline that was trained.
Treating disagreement as guaranteed uncertainty
Members can share blind spots because they use the same data and assumptions. Validate whether disagreement predicts errors on representative held-out and shifted data before using it for automated risk decisions.
When to use an ensemble
Ensembling is attractive when predictive quality has high value, several competent models make meaningfully different errors, and the system can afford additional training and inference cost. It is especially practical for offline scoring, competitions, high-value decisions with manageable request volume, or systems where parallel hardware is already available.
A single model is often preferable when latency, memory, energy, or operational simplicity dominates; when candidate models make almost identical errors; or when the measured quality gain is too small to justify multiple artifacts.
Before building a complex stack, test the simplest ensemble that could work:
train independent competent models
align their outputs
average probabilities or regression predictions
evaluate on untouched data
measure end-to-end serving costOnly add weighting or stacking when the validation evidence supports the added complexity.
Conclusion
Ensembling works by combining competent models whose errors are not perfectly aligned. Averaging can cancel some model-specific error, but model count alone does not create that benefit.
Start with equal-weight prediction averaging, inspect whether members actually contribute complementary information, and evaluate the ensemble on data that was not used to choose its composition. Then measure the operational side of the trade-off: latency, memory, compute, and maintenance.
A useful ensemble is not the one with the most models. It is the smallest combination whose measured quality improvement is worth its additional cost.