Neural Tangent Kernel

Understand Wide Neural Networks with the Neural Tangent Kernel

Understand Wide Neural Networks with the Neural Tangent Kernel A neural network may contain millions of parameters, yet a useful theoretical view asks a smaller question: when one training example changes the parameters, how does that update affect the prediction for another example? The neural tangent kernel (NTK) answers that question through gradients. It measures how similarly two inputs respond to an infinitesimal parameter update. In a particular infinite-width regime, this kernel becomes effectively fixed during training, turning a nonlinear parameter-optimization problem into a much simpler kernel process in function space.