Early Exit

Cut Neural Network Inference Cost with Early Exits

Cut Neural Network Inference Cost with Early Exits A neural network usually spends the same depth of computation on every input. That is convenient, but not every input needs the same effort. A clear image of a stop sign may be classified correctly after relatively shallow processing, while an occluded sign may need the full network. Early-exit inference adds intermediate prediction points to a model and lets sufficiently confident inputs stop before the final layer. The aim is not to make every request cheaper. It is to spend less computation on easier cases while preserving a deeper path for harder ones.