Cut Neural Network Inference Cost with Early Exits

A neural network usually spends the same depth of computation on every input. That is convenient, but not every input needs the same effort. A clear image of a stop sign may be classified correctly after relatively shallow processing, while an occluded sign may need the full network.

Early-exit inference adds intermediate prediction points to a model and lets sufficiently confident inputs stop before the final layer. The aim is not to make every request cheaper. It is to spend less computation on easier cases while preserving a deeper path for harder ones.

This article develops a practical mental model for early exits, shows how confidence gates work, and explains the calibration, threshold, latency, and evaluation issues that determine whether the technique is useful in production.

Think of model depth as a compute budget

Consider a classifier with twelve sequential blocks. Ordinary inference runs all twelve:

input
  |
block 1
  |
...
  |
block 12
  |
classifier

An early-exit model might attach classifiers after blocks 4, 8, and 12:

input
  |
blocks 1-4 -> exit A
  |
blocks 5-8 -> exit B
  |
blocks 9-12 -> final exit

At exit A, the system asks whether the current prediction is reliable enough to return. If not, computation continues to exit B. Inputs that remain uncertain use the full model.

The important idea is conditional depth. The network has a maximum depth, but each input may consume a different fraction of it.

This changes the cost model. If a request exits after block 4, blocks 5 through 12 do not need to run for that request. If most requests reach the final exit, however, the extra intermediate classifiers and gating logic can add overhead without saving much work.

Start with a simple confidence gate

Suppose an intermediate classifier produces probabilities for three classes:

cat:  0.93
dog:  0.05
bird: 0.02

A simple policy can use the largest predicted probability:

if max_probability >= 0.90:
    return intermediate_prediction
else:
    continue_to_deeper_layers

This example demonstrates the mechanism, not a production recommendation. A value such as 0.93 is useful only if intermediate confidence has been evaluated on representative data.

Now consider a second input:

cat:  0.46
dog:  0.43
bird: 0.11

The gate keeps processing because the model has not separated the top candidates strongly enough.

A threshold creates a direct trade-off. Lowering it sends more requests through early exits, reducing average computation but increasing the chance of accepting weak intermediate predictions. Raising it is more conservative, so more requests reach deeper layers.

Intermediate predictions need their own evaluation

It is tempting to treat an early classifier as a smaller copy of the final classifier. In practice, it has a different input representation and usually a different error profile.

A shallow representation may already separate simple examples while lacking features required for subtle distinctions. An exit after block 4 can therefore be strong on common, visually obvious cases and weak on rare or ambiguous cases.

Evaluate each exit separately. Useful measurements include:

• task quality at that exit if every example stopped there;
• the fraction of requests accepted by a chosen gate;
• quality on the subset that the gate accepts;
• quality on the subset forwarded deeper;
• latency and compute consumed before the exit;
• final system quality after combining all exit decisions.

The accepted subset matters most. An intermediate exit does not need to match the final model on every input. It needs to be dependable on the inputs its gate chooses to accept.

This distinction prevents a common mistake: rejecting a useful early exit because its standalone accuracy is modest. If its gate reliably selects a high-quality subset, the exit can still reduce average cost.

Confidence is not the same as correctness

Softmax probabilities can be poorly calibrated. A classifier can assign 0.97 to predictions that are correct much less often than 97 percent. An early-exit policy that interprets raw probability as correctness can then stop too aggressively.

Calibration checks whether confidence aligns with observed outcomes. For example, among predictions near 0.9 confidence, roughly 90 percent should be correct under a well-calibrated interpretation for that population.

Calibration should be measured for each exit, not copied from the final classifier. The logits at block 4 and block 12 come from different representations and can have different confidence behavior.

A held-out calibration set can support post-training methods such as temperature scaling. For logits z and positive temperature T, the calibrated probabilities are:

p = softmax(z / T)

A larger T softens the distribution; a smaller T sharpens it. The temperature is fitted on held-out data rather than chosen because it produces a convenient exit rate.

Calibration does not make the classifier inherently more capable. It makes confidence more useful as a decision signal when the calibration assumptions remain reasonably valid.

A margin can be more useful than a raw maximum

Maximum probability is only one gating signal. Another simple option is the gap between the top two class scores.

Suppose two predictions have the same top probability:

case A: [0.80, 0.19, 0.01]
case B: [0.80, 0.10, 0.10]

The maximum is 0.80 in both cases, but the runner-up differs. A top-two margin captures that separation:

margin = highest_score - second_highest_score

For the examples above, the margins are 0.61 and 0.70.

Entropy is another option. A probability distribution concentrated on one class has lower entropy than a diffuse distribution. None of these signals is universally superior. Their value depends on how strongly they separate correct intermediate predictions from incorrect ones on the deployment distribution.

A gate can also use a small auxiliary model, but added complexity has a cost. If the gate itself consumes meaningful latency, memory bandwidth, or accelerator work, some of the benefit from exiting early disappears.

Choose thresholds from a system objective

Selecting a threshold by intuition makes the resulting trade-off hard to defend. A better approach is to sweep candidate thresholds on held-out data and measure the resulting system behavior.

For each threshold, record at least:

exit rate
task metric
mean latency
tail latency
average executed depth

Imagine a single early exit produces these illustrative results:

These numbers are a teaching example, not expected values for other models. They show the shape of the decision: a more permissive gate saves more depth but may give up more quality.

The right operating point comes from application constraints. A mobile vision feature may prioritize energy and median latency. A medical triage component may place a much tighter bound on error and use early exits only for narrowly defined cases, if at all.

Average compute does not guarantee lower latency

Skipping layers sounds equivalent to lower latency, but real serving systems add complications.

With batch size one, an early exit can avoid later kernels directly. With a batch of many requests, different examples may want to exit at different depths. If the implementation keeps the original batch shape and merely masks completed examples, later layers may still perform most of the same work.

Dynamic batching can regroup active examples, but regrouping introduces scheduling and memory movement. Accelerators also tend to perform better on sufficiently large, regular workloads. A smaller surviving batch can have worse hardware utilization even though it executes fewer mathematical operations.

Measure end-to-end latency on the target serving path. FLOP reduction is useful for analysis, but it is not a latency guarantee.

Tail latency also deserves separate attention. Early exits often improve the median more than the high percentile because difficult requests still traverse the full model. If the service-level objective is dominated by p99 latency, conditional depth may provide less value than its average numbers suggest.

Training an early-exit model

Intermediate classifiers need useful representations at their attachment points. A common training setup adds a loss for each exit:

L = w1 * L_exit1 + w2 * L_exit2 + w3 * L_final

The weights control how much each prediction head influences shared layers.

Giving shallow exits more weight can improve their predictions, but it can also change the features produced for deeper computation. The objectives are coupled because early blocks are shared. Treating the weights as harmless bookkeeping can therefore reduce final-exit quality.

Another approach is to train the main network first, freeze some or all of it, and then fit intermediate heads. This reduces interference with the final model but limits how much the shared representation can adapt to support early decisions.

Distillation can also guide an intermediate head using a deeper model’s output distribution. That can transfer useful class relationships, but it does not remove the need to validate the gate. A shallow head can imitate a teacher better and still be unreliable on a particular accepted subset.

Watch for distribution shift

An early-exit gate is a routing policy. Distribution shift can damage both the classifier and the routing decision.

Suppose a camera classifier was calibrated mostly on daylight images. At night, an intermediate head may remain confident on unfamiliar visual patterns even when its error rate rises. The gate can then send exactly the wrong cases through the cheap path.

Monitor exit rates as well as task metrics. A sudden change in the fraction of requests stopping at a shallow exit can reveal a population change even before aggregate accuracy is available.

When labels arrive slowly, useful operational signals include confidence distributions, exit-depth distributions, input-quality indicators, and disagreement between intermediate and final predictions on a sampled shadow path.

A conservative fallback is valuable. If the gate encounters an unsupported condition, missing feature, invalid score, or explicit uncertainty signal, continue to the deeper path rather than forcing an early decision.

Common mistakes

Using one threshold for every exit

A confidence value at one depth is not automatically comparable with the same value at another depth. Calibrate and tune each gate using its own outputs.

Optimizing only the exit rate

A high early-exit rate looks efficient but says nothing about accepted prediction quality. Track quality conditional on exiting.

Ignoring class-specific behavior

A global threshold can hide large differences between classes. If one class is systematically overconfident or safety-critical, per-class analysis may show that a shared gate is inappropriate.

Reporting theoretical compute only

Layer counts and FLOPs do not include batching effects, synchronization, gate overhead, data movement, or runtime scheduling. Benchmark the actual deployment stack.

Evaluating on the training distribution alone

A gate tuned tightly to one validation set can become brittle when traffic changes. Use representative held-out data and test plausible shifts that matter for the application.

When early exits fit

Early-exit inference is attractive when model depth is substantial, many inputs are genuinely easy, intermediate representations support useful predictions, and the serving stack can avoid later work for completed requests.

It is less attractive when almost every input needs full depth, intermediate heads add too much overhead, batching makes conditional execution inefficient, or errors from shallow decisions carry unacceptable consequences.

A simpler fixed-depth model can also be a better engineering choice. If a smaller model already meets the quality target, it avoids routing logic, multiple heads, calibration maintenance, and variable execution paths. Early exits are most compelling when the application benefits from retaining a strong deep path while exploiting a meaningful population of easy requests.

Build the gate as part of the model system

The useful unit is not an intermediate classifier by itself. It is the combination of prediction head, confidence signal, calibration procedure, threshold, fallback behavior, and serving runtime.

Start with one intermediate exit rather than many. Measure its accepted-subset quality and actual latency savings, then sweep the threshold against a concrete quality constraint. Add more exits only when measurements show that additional depth choices provide enough value to justify their complexity.

That approach keeps early-exit inference focused on its real purpose: allocating model computation according to input difficulty without treating confidence as a guarantee.