A classifier can report impressive accuracy while failing on the cases you care about most. This happens easily when one class is much more common than another.
Imagine a model that detects defective components on a production line. In a test set of 1,000 components, 950 are normal and 50 are defective. A model that predicts normal for every component is correct 95% of the time, yet it detects none of the defects.
The problem is not that accuracy was calculated incorrectly. Accuracy simply answers a question that ignores how errors are distributed across classes.
This article builds a practical mental model for evaluating imbalanced classifiers. You will learn what balanced accuracy and macro F1 measure, why they can disagree, and how to choose a metric that matches the failure you need to detect.
Start from the confusion matrix
For binary classification, evaluation becomes easier when you first count four outcomes for a chosen positive class:
predicted defective predicted normal
actually defective TP FN
actually normal FP TNTP means a defective component was detected. FN means a defect was missed. FP means a normal component triggered a false alarm. TN means a normal component was correctly accepted.
Ordinary accuracy is:
accuracy = (TP + TN) / (TP + TN + FP + FN)For the model that predicts every component as normal:
TP = 0
FN = 50
FP = 0
TN = 950
accuracy = 950 / 1000 = 0.95The number is correct, but the majority class dominates it. Each example contributes equally, so the 950 normal examples overwhelm the 50 defective examples.
When class frequencies are uneven, ask a second question: how well does the model perform within each class?
Balanced accuracy gives each class equal influence
Recall measures the fraction of examples from a class that the classifier identifies correctly. For the defective class:
defect recall = TP / (TP + FN)For the normal class, treating normal as the class of interest gives:
normal recall = TN / (TN + FP)In binary classification, balanced accuracy is the average of these two recalls:
balanced accuracy = (defect recall + normal recall) / 2For the always-normal classifier:
defect recall = 0 / 50 = 0.00
normal recall = 950 / 950 = 1.00
balanced accuracy = (0.00 + 1.00) / 2 = 0.50Now the failure is visible. The model is perfect on one class and useless on the other, so its balanced accuracy is 0.50 rather than 0.95.
The same idea extends naturally to multiclass classification: calculate recall separately for every class, then take their unweighted mean. A rare class therefore contributes as much to the final metric as a common class.
This weighting is useful when class-level coverage matters. It is also a deliberate choice. If class prevalence represents real operational importance, giving every class equal weight may not match the cost of mistakes in production.
Macro F1 also accounts for false alarms
Balanced accuracy focuses on recall. It tells you whether examples from each class are being found, but it does not directly penalize a class for attracting many examples that belong elsewhere.
That is where precision becomes useful:
precision = TP / (TP + FP)
recall = TP / (TP + FN)The F1 score combines precision and recall using their harmonic mean:
F1 = 2 * precision * recall / (precision + recall)For macro F1, calculate F1 independently for each class and then take the unweighted mean:
macro F1 = (F1_class_1 + F1_class_2 + ... + F1_class_k) / kBecause every class receives equal weight, a weak minority class cannot disappear behind a large majority class. Unlike balanced accuracy, however, each class’s F1 also reflects false-positive predictions for that class.
Consider a three-class classifier with these per-class results:
class precision recall
normal 0.98 0.95
scratch 0.40 0.80
crack 0.90 0.45The scratch class has high recall but low precision: most real scratches are found, but many non-scratches are incorrectly labelled as scratches. The crack class shows the opposite pattern: predictions of cracks are usually right, but many real cracks are missed.
Balanced accuracy sees the recall difference. Macro F1 also captures the precision problem for scratch. Neither metric tells the whole operational story by itself, but they expose different failure modes.
Why balanced accuracy and macro F1 can disagree
Suppose two models have similar per-class recall. Their balanced accuracy can therefore be similar. If one model produces many more false positives for a minority class, its precision for that class falls and its macro F1 can be much lower.
That disagreement is informative rather than contradictory:
balanced accuracy -> Are examples from every class being found?
macro F1 -> For every class, are recall and precision both reasonable?This distinction matters in systems where false alarms consume resources. A defect detector that catches nearly every defect may have strong recall, but if it flags half the production line for manual inspection, recall alone does not describe the operational burden.
Conversely, F1 deliberately ignores true negatives for the class whose F1 is being calculated. In a binary task where correctly rejecting negatives has direct value, you may need additional metrics or an explicit cost calculation.
Do not confuse macro, micro, and weighted averaging
Libraries often expose several averaging modes for multiclass metrics. They answer different questions.
Macro averaging computes a metric for each class and gives every class equal weight. This makes weak performance on a rare class visible.
Weighted averaging also computes the metric per class, but weights each result by the number of true examples in that class. Majority classes therefore have more influence. This can be useful when you want a class-aware metric that still reflects the observed class distribution, but it can again make rare-class failures less prominent.
Micro averaging aggregates the underlying decisions across classes before calculating the metric. In ordinary single-label multiclass classification, micro precision, micro recall, and micro F1 collapse to accuracy because every wrong prediction creates one incorrect class assignment and every correct prediction creates one correct assignment.
That last property does not hold for every problem formulation. Multilabel classification, for example, allows several labels per example, so micro F1 and subset accuracy answer different questions.
Always record the averaging mode with the metric name. Reporting only F1 = 0.91 is incomplete when multiple averaging definitions are possible.
Evaluate the classes you will actually encounter
A metric cannot repair an unrepresentative evaluation set.
Suppose defects in production include hairline cracks under low light, but your test set contains only large, well-lit cracks. A high macro F1 on that test set does not establish performance on the missing condition. The metric summarizes the examples it receives; it cannot measure absent cases.
Class imbalance also affects uncertainty in the metric itself. If a test set contains only ten examples of a rare class, one additional mistake changes that class’s recall by ten percentage points. Because macro metrics give that class equal weight, the overall score can move noticeably from a small number of outcomes.
For important rare classes, inspect the raw support count alongside the metric:
class examples precision recall F1
normal 9500 0.99 0.98 0.98
scratch 420 0.81 0.76 0.78
crack 18 0.75 0.67 0.71The crack F1 may be useful as an estimate, but 18 examples provide much less evidence than 9,500 examples do for the normal class. Collecting more representative evaluation data may be more valuable than comparing another decimal place between models.
Keep threshold selection separate from final evaluation
Binary classifiers often produce scores or probabilities rather than final labels. Precision, recall, F1, balanced accuracy, and ordinary accuracy all depend on the threshold used to turn those scores into labels.
If you try many thresholds on the test set and publish the best result, the test set has become part of model selection. The reported score can then be optimistic because the threshold was chosen to fit that data.
A cleaner workflow is:
training data -> fit model
validation data -> choose threshold and other decisions
test data -> estimate final performance once decisions are fixedThe metric used to choose the threshold does not have to be the only metric you report. For example, you might select a threshold that achieves a required defect recall, then report precision, balanced accuracy, macro F1, and the confusion matrix at that threshold.
Match the metric to the cost of errors
Neither balanced accuracy nor macro F1 is automatically the right production objective.
Suppose missing a dangerous defect costs far more than sending a normal component to manual review. Averaging class recall equally does not encode that asymmetry, and maximizing macro F1 may trade away defect recall to improve precision.
In that situation, a more useful evaluation might require a minimum recall for the dangerous class and then optimize another quantity among models that satisfy it. If you can estimate operational costs, expected cost can be even more direct:
expected cost =
false_negatives * cost_per_missed_defect
+ false_positives * cost_per_false_alarmThis simplified expression assumes those costs are reasonably stable per event. Real systems may have nonlinear costs, capacity limits, delayed outcomes, or different costs for different subtypes.
The broader lesson is that imbalance is not itself the business objective. It is a warning that aggregate metrics can conceal class-specific behavior.
Common evaluation mistakes
A few mistakes repeatedly make imbalanced-classification results look stronger or clearer than they are.
Reporting accuracy alone
Accuracy is still useful, but it should not be the only view when important classes have very different frequencies. Pair it with class-level results and a metric that prevents the majority class from dominating the summary.
Reporting only a macro score
Macro averaging exposes weak classes, but the average can hide which class failed. Keep the per-class precision, recall, F1, and support counts available during model development.
Treating F1 as a probability-quality metric
F1 evaluates thresholded decisions. It does not tell you whether a predicted probability of 0.8 behaves like an 80% probability. Probability calibration is a separate property.
Comparing scores from different class sets
Macro metrics depend on which classes are included. If one evaluation excludes an unsupported class while another includes it, their macro scores are not directly comparable without explaining the difference.
Ignoring the deployment distribution
Equal class weighting can be excellent for diagnosing minority-class behavior while still being a poor estimate of average production error. Use diagnostic metrics and operational metrics for the questions they actually answer.
When to use each metric
Use ordinary accuracy when each example-level mistake has roughly comparable importance and the observed class distribution is meaningful for your objective. It remains a useful baseline even when you report other metrics.
Use balanced accuracy when you want every class’s recall to influence the summary equally. It is especially interpretable when the main concern is whether the classifier detects examples from each class.
Use macro F1 when every class deserves equal attention and both missed examples and false-positive assignments matter. It is often useful for comparing multiclass classifiers whose class frequencies differ substantially.
Use per-class metrics or explicit costs when particular classes have distinct safety, financial, or operational consequences. A single scalar average cannot express every asymmetric requirement.
In practice, a compact evaluation table plus the confusion matrix is often more informative than trying to find one universal score.
Conclusion
Class imbalance makes ordinary accuracy easy to misread because majority-class examples can dominate the result. Balanced accuracy counters that effect by averaging recall across classes. Macro F1 also gives each class equal weight, while incorporating both precision and recall for each class.
The useful question is not which metric is universally better. Ask what kind of failure the metric would reveal. If missing a class is the main concern, balanced accuracy provides a direct signal. If false alarms and misses both matter across classes, macro F1 adds information. For consequential decisions, keep the per-class results visible and connect evaluation to the actual cost of errors.