Understand Wide Neural Networks with the Neural Tangent Kernel

A neural network may contain millions of parameters, yet a useful theoretical view asks a smaller question: when one training example changes the parameters, how does that update affect the prediction for another example?

The neural tangent kernel (NTK) answers that question through gradients. It measures how similarly two inputs respond to an infinitesimal parameter update. In a particular infinite-width regime, this kernel becomes effectively fixed during training, turning a nonlinear parameter-optimization problem into a much simpler kernel process in function space.

That result is useful as a mental model even when a real network is finite. It clarifies how examples interact through gradients, what the wide-network approximation assumes, and where the approximation stops describing practical neural networks well.

Start with prediction gradients

Consider a scalar-output network:

f(x; theta)

Here, x is an input and theta is the vector containing every trainable parameter.

For a fixed input, take the gradient of the output with respect to the parameters:

g(x) = grad_theta f(x; theta)

This gradient is a vector with one component per parameter. It describes the local sensitivity of the prediction to a tiny parameter change.

Now compare two inputs, x and z, by taking the dot product of their prediction gradients:

K(x, z) = g(x) dot g(z)

That quantity is the neural tangent kernel at the current parameter values.

The core intuition is concrete. If g(x) and g(z) point in similar directions, a small parameter update that raises the output for x tends to raise the output for z as well. If their dot product is negative, the same local update tends to move the two outputs in opposite directions. A value near zero means their first-order responses interact weakly.

This is not ordinary similarity between the raw inputs. Two inputs can look close in input space yet have very different prediction gradients. The NTK measures similarity through the network’s current parameterization.

Follow one gradient step

Suppose the network is trained on one example (x, y) with squared error:

L = 1/2 * (f(x; theta) - y)^2

Define the residual:

r = f(x; theta) - y

The loss gradient is:

grad_theta L = r * g(x)

With a small gradient-descent step of size eta:

delta_theta = -eta * r * g(x)

What happens to the prediction at another input z?

For a sufficiently small step, a first-order approximation gives:

delta_f(z) ~= g(z) dot delta_theta
           = -eta * r * K(z, x)

This equation exposes the role of the kernel. Training on x changes the prediction at z in proportion to K(z, x).

For a tiny numerical example, suppose:

f(x) = 1.4
y    = 1.0
r    = 0.4
eta  = 0.1
K(z, x) = 2.0

Then:

delta_f(z) ~= -0.1 * 0.4 * 2.0
           ~= -0.08

The update made to reduce the error at x also lowers the output at z by about 0.08 under the local approximation.

This is the simplest useful NTK picture: training examples influence other predictions according to gradient similarity.

Extend the picture to a dataset

For training inputs x1, x2, ..., xn, compute the kernel between every pair. The resulting matrix is the NTK Gram matrix:

K[i, j] = K(xi, xj)

For scalar outputs, it has shape n x n.

Under continuous-time gradient descent, often called gradient flow, and squared error, the prediction vector on the training set follows a particularly clean equation when the kernel is treated as fixed:

df/dt = -K (f - y)

The vector f - y contains the current residuals. Multiplication by K mixes those residuals according to gradient similarity.

If K is symmetric positive semidefinite, it can be decomposed into eigenvectors and nonnegative eigenvalues. Along an eigenvector with eigenvalue lambda, the residual decays proportionally to:

exp(-lambda * t)

up to the convention used for the training-rate scale.

Large-eigenvalue directions therefore decay more quickly under this idealized fixed-kernel dynamics. Small-eigenvalue directions change more slowly. This connects optimization speed to the geometry encoded by the network’s prediction gradients.

The statement is about the stated squared-error gradient-flow setting. Discrete optimizer steps, momentum, adaptive optimizers, other losses, regularization, and finite width can change the dynamics.

The infinite-width result changes the problem

For an ordinary finite network, the parameters move during training. Because g(x) depends on those parameters, the NTK generally changes too:

K_t(x, z) = grad_theta f(x; theta_t)
            dot
            grad_theta f(z; theta_t)

The major theoretical result associated with the NTK concerns a suitable infinite-width limit. Under specific architecture, parameterization, initialization, and training assumptions, the NTK converges to a limiting kernel and stays constant during training.

That matters because a fixed kernel makes the network’s function-space dynamics tractable. Instead of tracking a huge nonlinear trajectory through parameter space, the analysis can track predictions through a kernel matrix.

The phrase infinite width is not shorthand for “any large model.” It is a mathematical limit. The exact result depends on how widths grow, how weights are scaled, and which training setup is used. A finite production network may resemble the limiting dynamics to some degree without matching them exactly.

This distinction prevents a common mistake: the NTK limit is a model of a training regime, not a universal description of every neural network.

Feature movement separates two regimes

A useful way to interpret the NTK is to contrast nearly fixed features with substantial feature movement.

Suppose a network’s hidden representations change only a little during training. Its prediction gradients may also remain close to their initial values. A fixed kernel can then provide a useful approximation of how examples influence one another. This behavior is often associated with a kernel regime or lazy-training regime.

Now consider a network whose internal representations change substantially. The model is not merely fitting coefficients on top of an almost fixed local geometry; it is changing the geometry itself. The kernel can evolve as the parameters move.

That difference is important in practice. A fixed NTK can explain optimization behavior that arises from the initial parameterization, but it cannot fully capture benefits that depend on substantial representation change.

You can express the distinction as:

nearly fixed NTK:
initial gradient geometry -> mostly preserved -> predictions fit targets

changing NTK:
initial gradient geometry -> changes during training -> new interactions emerge

Neither regime is automatically preferable. The useful question is which approximation describes the network and training setup being studied.

Use the empirical NTK as a diagnostic

You do not need an infinite network to compute an NTK. For a finite network, the gradient definition gives an empirical NTK at a particular parameter state.

For a small probe set, a conceptual procedure is:

for each input xi:
    gi = gradient of model output with respect to parameters

for each pair (xi, xj):
    K[i, j] = dot(gi, gj)

For vector-valued outputs, the kernel has additional output dimensions unless those dimensions are aggregated under a chosen definition. Be explicit about that choice when comparing implementations.

The empirical kernel can support several diagnostics.

First, compare K at initialization with K later in training. A small change suggests that fixed-kernel reasoning may be more relevant to that run. A large change signals that representation or gradient geometry has moved enough that an initialization-only kernel misses part of the story.

Second, inspect the spectrum of the Gram matrix on a fixed probe set. Very small eigenvalues indicate directions in prediction space that fixed-kernel gradient flow would fit slowly. The spectrum can also reveal strong redundancy among gradient responses.

Third, compare kernel similarity across groups of examples. If examples from two groups have consistently strong cross-group kernel values, updates from one group can have substantial first-order effects on the other. That observation can motivate closer checks for interference or transfer.

These are diagnostics, not guarantees about generalization. The result depends on the probe set, parameter state, output definition, and numerical method.

Computing the full kernel can be expensive

The definition is simple, but direct computation can be costly.

If a model has P parameters and the probe set has N inputs, explicitly materializing every prediction-gradient vector can require storage proportional to N * P. Building all pairwise dot products also creates an N x N matrix.

For large models or datasets, that can dominate memory and compute.

Practical analysis therefore often uses a small representative probe set, Jacobian-vector products, vector-Jacobian products, structured kernel routines, or approximations rather than materializing the complete Jacobian.

The right method depends on the question. If you only want to compare kernel drift at several checkpoints, a fixed sample of a few dozen or few hundred inputs may be more informative and far cheaper than attempting a dataset-wide matrix.

Also keep numerical precision in mind. Kernel matrices can be ill-conditioned, and tiny eigenvalues are especially sensitive to floating-point error. Treat near-zero spectral values cautiously rather than assigning strong meaning to every decimal place.

Common interpretation mistakes

The NTK is compact enough that several incorrect shortcuts are tempting.

Treating kernel similarity as semantic similarity. The kernel compares prediction gradients. It can correlate with semantic structure, but that is not guaranteed by the definition.

Assuming the kernel is fixed for every wide network. Constancy is an asymptotic result under particular scaling and training assumptions. Finite width alone does not guarantee a constant empirical kernel.

Equating a good NTK prediction with a complete account of training. If fixed-kernel dynamics match a run, that says the local gradient geometry explains much of the observed function movement. It does not establish that every internal mechanism is captured.

Ignoring parameterization. Two networks that compute similar functions can have different parameter scalings and therefore different tangent kernels. NTK statements must be tied to the parameterization used.

Using the training-set kernel as a direct generalization score. Training dynamics and out-of-sample behavior are related but distinct. A spectrum that looks convenient for optimization does not by itself prove strong performance on unseen data.

Computing a huge matrix without a decision in mind. Kernel analysis is most useful when it tests a concrete question, such as whether the tangent geometry remains stable or whether two groups interact strongly.

When the NTK view is useful

The NTK is especially useful when you want a principled bridge between neural-network optimization and kernel methods.

It can help with theoretical analysis of sufficiently wide networks, controlled experiments on training dynamics, comparisons between parameterizations, and diagnostics of gradient geometry. It also provides a precise baseline: if a finite network behaves very differently from its fixed initial kernel, that gap itself is evidence that the network is doing something beyond the fixed-kernel approximation.

A simpler tool is often preferable for routine model debugging. If the problem is exploding gradients, inspect gradient norms. If validation quality is poor, start with data quality, evaluation design, optimization settings, and ordinary error analysis. Computing an NTK should answer a specific question that those cheaper checks cannot.

The framework is also less informative when the behavior of interest depends heavily on large representation changes, discrete architectural decisions, non-gradient updates, or training procedures far from the assumptions behind the analysis.

Keep the mental model local

The most reusable NTK idea does not require taking width to infinity.

For each input, the network has a prediction gradient. The dot product between two such gradients tells you how strongly a tiny parameter update for one input is expected to affect the other input at first order. Collect those interactions across a dataset and you get a kernel matrix that describes local function-space training geometry.

The infinite-width theory adds a special case in which that geometry becomes fixed, making the dynamics much easier to analyze. Real networks can depart from that case, sometimes substantially.

When using the NTK, keep those two levels separate: the empirical kernel is a measurable property of a finite network at a given parameter state; the constant limiting kernel is a theoretical object obtained under additional assumptions. That distinction turns the NTK from an abstract formula into a practical tool for reasoning about how gradient-based training couples predictions.