Reduce Neural Network Inference Cost with Early Exits

A deep neural network normally applies every block to every input, even when an intermediate representation already supports a confident prediction. Early-exit inference changes that fixed-depth path. It attaches prediction heads at intermediate points and lets selected inputs stop before the final block.

The appeal is conditional computation: easy cases can consume less compute while ambiguous cases retain access to the full network. The difficult part is deciding when an intermediate prediction is reliable enough to return. A poor exit policy can save computation by silently moving errors toward the shallow heads.

Fixed depth spends the same path on unequal inputs

Consider an image classifier with twelve sequential blocks. One input contains a centered, unobstructed object. Another contains a small object behind clutter. A conventional network sends both through all twelve blocks:

input A -> 1 -> 2 -> 3 -> ... -> 12 -> classifier
input B -> 1 -> 2 -> 3 -> ... -> 12 -> classifier

The network does not inspect the intermediate state and decide that input A has already become straightforward.

An early-exit design might add classifiers after blocks 4 and 8:

input -> blocks 1-4 -> head A -> exit or continue
                         |
                         v
                    blocks 5-8 -> head B -> exit or continue
                                      |
                                      v
                                 blocks 9-12 -> final head

Each intermediate head maps the representation available at that depth to the task output. An exit policy then decides whether to return that output or pay for more computation.

This is not the same as removing blocks from the model. Pruning or using a smaller model changes the computation available to every request. Early exits keep deeper computation available but invoke it selectively.

An exit rule turns confidence into compute allocation

For a classifier, the simplest policy uses the largest predicted class probability. Suppose the head after block 4 emits:

class A: 0.94
class B: 0.04
class C: 0.02

With an exit threshold of 0.90, the request stops at block 4. If another input produces:

class A: 0.46
class B: 0.43
class C: 0.11

it continues.

For predicted probabilities (p_1, \ldots, p_K), this policy can be written as:

[ \max_k p_k \ge \tau ]

where (\tau) is the threshold for that exit. Raising (\tau) generally sends more inputs deeper. Lowering it generally increases the exit rate.

The threshold is a routing control, not a proof that the prediction is correct. A softmax value of 0.94 does not by itself guarantee a 94% chance of correctness. Neural classifiers can be miscalibrated, and calibration can differ across intermediate heads, classes, and input populations.

Other policies can use predictive entropy, the margin between the top two classes, or a separate gating model. The same constraint applies: the routing signal must be evaluated against actual outcomes. A score that appears decisive can still be systematically wrong.

Intermediate heads need useful representations

Adding a classifier to an arbitrary block does not make that block suitable for an exit. Earlier representations may encode local or low-level features while the final task requires information that emerges only after additional computation.

Training therefore matters. A model designed for early exits commonly includes losses for intermediate heads as well as the final head. A simplified objective is:

[ L = \lambda_1 L_1 + \lambda_2 L_2 + \lambda_3 L_3 ]

Here, (L_1) and (L_2) are losses from intermediate heads, (L_3) is the final-head loss, and the (\lambda) values control their contribution.

Those weights create a real optimization choice. Giving shallow heads more influence can make intermediate predictions more useful, but it can also alter the representations optimized for deeper computation. The result depends on the architecture, task, training procedure, and weighting scheme; an auxiliary head is not automatically free in accuracy terms.

A separate approach trains exit heads after the backbone has already been trained, leaving backbone parameters fixed. That avoids changing the backbone but restricts each exit to information already present at its attachment point. If the intermediate representation does not separate the target classes well, a larger exit head may add cost without fixing the underlying limitation.

Average depth is more informative than exit rate alone

An exit rate sounds useful but does not directly state the compute reduction. Exiting 60% of requests after 90% of the network saves much less work than exiting the same fraction after 30%.

For a model with exit depths (d_1, d_2, \ldots, d_m), let (q_i) be the fraction of inputs that finish at depth (d_i). A simple normalized depth estimate is:

[ D = \sum_i q_i d_i ]

If depth is expressed as a fraction of the full backbone, consider:

50% exit at depth 0.40
30% exit at depth 0.70
20% reach depth 1.00

Then:

[ D = 0.50(0.40) + 0.30(0.70) + 0.20(1.00) = 0.61 ]

The average request traverses 61% of the backbone depth under this simplified model.

That does not imply a 39% latency reduction. Exit heads have their own cost. Some operations do not scale linearly with depth. Hardware utilization, memory traffic, batching, synchronization, and framework overhead can dominate portions of inference. The depth calculation is useful for reasoning about routing, while measured latency and resource consumption remain the relevant deployment quantities.

Batching changes the latency picture

Conditional depth is easiest to exploit when requests can diverge without forcing each other to wait. Large accelerator batches complicate that property.

Suppose half the sequences in a batch qualify for an early exit while the other half continue. A serving implementation may need to compact the surviving sequences into a new batch. That operation has overhead and can reduce accelerator utilization. If the system instead keeps exited items in the batch until every item finishes, much of the intended compute saving disappears.

This creates a difference between per-input computation and system throughput. Early exits can reduce the former without producing a proportional improvement in the latter.

The deployment architecture therefore belongs in the evaluation. On a latency-sensitive service with small or dynamic batches, conditional depth may map naturally to execution. On a throughput-oriented accelerator service built around large static batches, a smaller fixed-depth model can be operationally simpler and may use the hardware more consistently.

Thresholds belong on held-out data

Choosing a threshold from a few successful examples makes the routing policy hard to defend. The useful object is a curve showing task quality against computation or latency over held-out data.

For each candidate threshold, record at least:

• task metric at the system output;
• fraction of inputs using each exit;
• average executed depth or measured compute;
• end-to-end latency under representative load.

A single global accuracy number can also hide where early exits fail. If some classes or input groups receive overconfident shallow predictions, their error rate can rise even when aggregate accuracy changes little. Per-class or slice-level checks are appropriate when those distinctions matter to the application.

Thresholds should be selected using validation data, not the final test set. The test set is most useful as an independent estimate after model and routing choices have been fixed.

Calibration and distribution shift affect routing

An early-exit policy depends on more than the ranking quality of the classifier. It depends on the relationship between its confidence signal and its errors.

Suppose an intermediate head becomes overconfident on blurred images. A confidence threshold can then do the opposite of the intended behavior: difficult blurred inputs exit early because the head assigns an unjustifiably high score. Sending only low-confidence inputs deeper does not help if confidence itself fails to identify the risky cases.

Calibration methods can improve the relationship between scores and empirical outcomes on representative data, but calibration is not permanent. A changed input distribution can alter both model accuracy and the exit pattern.

Production monitoring should therefore include routing statistics as well as final predictions. A sudden increase in shallow exits, a shift in confidence distributions, or a class-specific change in exit depth can indicate that the policy is operating outside the conditions used to set its thresholds.

Early exits fit some tasks better than others

Classification provides a clean fit because an intermediate head can produce a complete candidate answer at a known point. Other tasks need more care.

For token-generating models, “exit” can mean reducing the depth used to compute token representations rather than returning an entire response at an intermediate block. Generation is autoregressive, so a depth decision can interact with later tokens and cached states. Techniques in that setting require architecture-specific handling and should not be treated as a direct copy of classifier early exiting.

Dense prediction tasks also add constraints. In segmentation, for example, an intermediate output must have adequate spatial detail and the exit criterion must reflect uncertainty across many output positions. A single maximum class score is usually too crude to characterize an entire mask.

There are also cases where conditional depth offers little value. If nearly every valid input requires the deepest blocks to meet the target metric, intermediate heads add parameters and routing complexity without meaningful savings. If a compact fixed model already satisfies the quality target, deploying that model can be simpler than maintaining several exits and threshold policies.

Evaluate the system that will actually run

Early-exit inference is most useful when model difficulty varies across inputs and the serving stack can turn skipped depth into measurable resource or latency savings. Both conditions matter.

The decisive experiment is not whether an intermediate head can classify easy examples. It is whether a validated routing policy preserves the required task quality while improving a deployment metric that matters under representative traffic. That evaluation should include calibration, input slices, batch behavior, exit overhead, and the final execution environment.

Conditional computation creates an extra control surface between model quality and inference cost. Treating that control as part of the deployed model, rather than as a confidence shortcut, makes its limits much easier to see.