Activation Functions

Diagnose and Prevent Dying ReLU Units

ReLU is one of the simplest neural-network activation functions: negative inputs become zero and positive inputs pass through unchanged. That simplicity makes optimization efficient, but it creates a failure mode that can quietly waste model capacity. A unit can move into a state where its pre-activation is negative for every relevant training example, so its ReLU output stays zero and the unit stops receiving a useful gradient through that activation.