A deep neural network can fail before learning has had a fair chance. If its initial weights make activations or gradients shrink layer after layer, useful signals can become tiny. If those quantities grow too much, training can become unstable. The optimizer may receive a problem that is unnecessarily difficult even though the architecture and data are otherwise reasonable.

Weight initialization tries to start the network in a numerically useful regime. Two common schemes are Xavier initialization, also called Glorot initialization, and He initialization, also called Kaiming initialization. Both choose the scale of random weights from the size of a layer, but they make different assumptions about how signals pass through the activation function.

This article builds the idea from a simple variance calculation. You will learn what fan_in and fan_out mean, why Xavier and He use different scales, how to choose between them, what these formulas do not guarantee, and how to diagnose initialization problems in a real model.

Initialization is about signal scale, not lucky weights

Consider one neuron with n inputs:

z = w1*x1 + w2*x2 + ... + wn*xn

The neuron computes a weighted sum z, then usually passes it through an activation function.

If every weight starts extremely small, the weighted sum tends to be small. Repeating that behavior across many layers can make activations and backpropagated gradients progressively weaker. If weights start too large, the opposite problem can occur: intermediate values can grow, and saturating nonlinearities can also move into regions where their derivatives are very small.

A useful initialization therefore does not try to guess the final weights. It chooses a random starting scale that helps information travel through the network before training has substantially changed the parameters.

The central mental model is:

layer width + activation behavior
              |
              v
choose initial weight variance
              |
              v
keep forward and backward signals at workable scales

The formulas behind Xavier and He initialization come from simplified statistical assumptions. They are design rules, not guarantees that every architecture will preserve exactly the same variance.

Start with the variance of a weighted sum

Suppose the inputs x_i and weights w_i are independent, centered around zero, and identically distributed within the layer. Under these simplifying assumptions,

z = sum(w_i * x_i), for i = 1 ... n

has approximately

Var(z) = n * Var(w) * Var(x)

where n is the number of inputs feeding the neuron.

That n matters. If the weight variance stays fixed while the layer becomes wider, the variance of the weighted sum grows with the number of inputs.

For example, imagine inputs with variance 1 and 100 incoming connections. If the weights also had variance 1, the weighted sum would have variance around 100 under the assumptions above. A deeper network could amplify that scale repeatedly.

Instead, choose

Var(w) approximately 1 / n

and the weighted sum has variance around the input variance:

Var(z) approximately n * (1 / n) * Var(x)
       approximately Var(x)

This is the basic idea behind variance-scaled initialization. The activation function determines what adjustment is needed after the weighted sum.

Fan-in and fan-out describe layer connectivity

Initialization formulas usually refer to two quantities:

  • fan_in: the number of inputs contributing to one output unit;
  • fan_out: the number of output units receiving values from one input unit.

For a dense layer with weight matrix shaped as fan_out x fan_in, these names are literal.

Suppose a layer maps 256 input features to 128 output features:

fan_in  = 256
fan_out = 128

Convolutional layers use the same idea but include the receptive-field size. Libraries normally calculate these values for their own layer layouts, which is preferable to manually assuming a matrix orientation.

Why are there two numbers? Forward propagation is naturally influenced by how many inputs are summed into each output, while backward propagation is influenced by how gradients are distributed through connections in the reverse direction. Some initialization schemes balance both considerations rather than preserving only the forward signal.

Xavier initialization balances linear or symmetric activations

Xavier initialization was designed to keep signal variance from changing too aggressively across layers while considering both fan_in and fan_out.

A common Xavier normal initialization samples zero-mean weights with variance

Var(w) = 2 / (fan_in + fan_out)

so the standard deviation is

sqrt(2 / (fan_in + fan_out))

A common Xavier uniform form samples from

[-a, a]

with

a = sqrt(6 / (fan_in + fan_out))

These forms have the same variance because a uniform random variable on [-a, a] has variance a^2 / 3.

For the earlier dense layer with fan_in = 256 and fan_out = 128, Xavier normal gives

standard deviation = sqrt(2 / 384)
                   approximately 0.0722

The important point is not the decimal value. The scale automatically becomes smaller for layers with more connections.

Xavier initialization is a natural baseline for linear layers and for activations whose behavior is reasonably compatible with its variance assumptions, historically including tanh. It is not specifically designed for the signal loss introduced by a ReLU.

ReLU changes the variance calculation

A ReLU computes

ReLU(z) = max(0, z)

For a roughly symmetric zero-centered input distribution, about half of the pre-activations are negative and become zero. This changes the second moment of the signal. If initialization used the same scale as a purely linear transformation, repeated ReLU layers could reduce signal magnitude more than intended.

He initialization compensates by using a larger weight variance. For a ReLU network, the common fan-in form is

Var(w) = 2 / fan_in

or, for a normal distribution,

standard deviation = sqrt(2 / fan_in)

With fan_in = 256, that becomes

standard deviation = sqrt(2 / 256)
                   approximately 0.0884

That is larger than the Xavier scale in the previous example because it compensates for the gating effect of ReLU.

A common uniform He initialization with the same target variance uses

a = sqrt(6 / fan_in)
weights ~ Uniform(-a, a)

Again, the distribution shape differs, but the target variance is the central idea.

Choose the scheme to match the activation and objective

A practical starting rule is:

linear or tanh-like layer -> consider Xavier initialization
ReLU-like hidden layer    -> consider He initialization

This is a starting point, not a universal mapping. Modern architectures may include residual paths, normalization layers, gated activations, attention blocks, custom parameterizations, or deliberately scaled residual branches. Those details can change the desirable initialization behavior.

For leaky ReLU, the negative side is not completely zero. If its negative slope is a, a commonly used gain factor accounts for that slope rather than treating the activation as an ordinary ReLU. Framework initialization utilities often expose the activation or its parameter so they can calculate the corresponding gain.

The important engineering rule is to match the initializer to the actual nonlinearity and tensor role. Copying one initializer across every parameter in a model can be wrong even when that initializer is sensible for the main hidden layers.

Biases usually do not need the same random scale

Weight matrices create the multiplicative accumulation that the fan-based formulas are designed to control. Biases play a different role:

z = W*x + b

For many ordinary dense and convolutional layers, initializing biases to zero is a reasonable default because the random weights already break symmetry between units.

This is different from setting all weights to zero. If every neuron in a layer starts with identical zero weights, neurons can receive identical gradients and remain symmetric, preventing them from learning distinct features. Random weight initialization breaks that symmetry.

Some architectures intentionally use nonzero bias initialization. For example, a model may initialize a gate or an output prior for a task-specific reason. Such choices are architectural decisions and should not be replaced mechanically with a generic zero-bias rule.

Initialization does not replace normalization or residual design

It is tempting to treat a good initializer as a complete solution to exploding or vanishing signals. It is not.

Initialization controls the starting point. During training, weights change. The distribution of activations also depends on the data, optimizer, learning rate, normalization, residual connections, architecture depth, and nonlinearities.

Batch normalization, layer normalization, and residual connections address related but different problems. A normalization layer transforms activations according to its own definition. A residual connection creates a direct path through which signals can travel. Neither makes initialization irrelevant, and initialization does not make either mechanism unnecessary.

Think of these techniques as interacting parts of the model rather than substitutes:

initialization -> useful starting scale
normalization  -> controls activations according to layer-specific rules
residual paths -> improve signal and gradient routes through depth
optimizer      -> changes parameters after initialization

A model should be evaluated as the complete system.

Inspect a network before the first optimizer step

You can test whether an initializer produces sensible behavior without waiting for a full training run. Pass a representative mini-batch through the randomly initialized model and inspect activation statistics by layer.

A simple diagnostic record might look like this:

layer       activation std
input       1.00
hidden 1    0.96
hidden 2    0.91
hidden 3    0.87
hidden 4    0.82

A gradual change is not automatically a problem. Real networks do not satisfy the independence assumptions exactly. But a pattern such as

1.00 -> 0.20 -> 0.03 -> 0.004

suggests that useful signal scale is collapsing rapidly before training begins.

Likewise,

1.00 -> 4.8 -> 21 -> 97

is a warning that activations are growing aggressively.

After a backward pass on a representative batch, inspect gradient norms or standard deviations across depth as well. If early layers consistently receive values many orders of magnitude smaller than later layers at initialization, investigate the activation, initializer, normalization, and architecture before simply increasing the learning rate.

The absolute values depend on the model and loss, so there is no universal acceptable standard deviation or gradient norm. Look for severe systematic trends and compare alternative initializations under the same batch and architecture.

Do not infer success from activation variance alone

Variance preservation is a useful mental model, but it simplifies several facts about real networks.

First, activations are not generally independent. Features become correlated, especially after training and across structured inputs.

Second, a distribution is not fully described by its variance. Two activation distributions can have the same variance but very different tails, sparsity, or saturation behavior.

Third, forward stability does not guarantee backward stability. An initialization can produce reasonable activation statistics while gradients still behave poorly through a particular architecture.

Fourth, trainability is not the same as final model quality. An initializer that makes the first steps numerically stable does not guarantee better generalization or a better final optimum.

For these reasons, initialization diagnostics should include both forward and backward behavior and ultimately a short controlled training comparison.

Common initialization mistakes

Using the same fixed standard deviation for every layer

A fixed value ignores layer width. A standard deviation that is harmless for a narrow layer can produce much larger weighted sums in a wide layer because more terms are added together.

Fan-scaled initialization adapts the variance to connectivity.

Using Xavier everywhere in a deep ReLU network

Xavier is not designed specifically to compensate for ReLU’s gating. In a plain stack of ReLU layers, He initialization is usually the more appropriate starting rule.

That does not mean every tensor in a ReLU-based architecture should receive He initialization. Output heads, embeddings, normalization parameters, and specialized residual projections may follow different conventions.

Reinitializing a pretrained model indiscriminately

Pretrained weights already contain learned structure. Applying a generic initializer to the entire model destroys that information.

When adding a new task head to a pretrained network, initialize the new parameters according to the model or framework’s intended convention while preserving pretrained parameters unless you explicitly intend to train from scratch.

Changing initialization and learning rate together

If two runs differ in both initialization and optimizer settings, you cannot tell which change caused the observed behavior. For diagnosis, vary one major factor at a time and keep the random seed, data batch, architecture, and training setup as controlled as practical.

Treating one successful seed as proof

Initialization is random. A single run can be unusually good or bad. When initialization is the variable under study, compare several seeds or at least verify that the observed pattern is not specific to one random draw.

When a simpler default is enough

You do not need to hand-calculate an initializer for every ordinary model. Mature neural-network libraries provide sensible defaults for common layers, and established architectures often come with initialization conventions that are part of the design.

Prefer the architecture’s documented convention when reproducing or fine-tuning a known model. Custom initialization is most useful when you are building a new architecture, replacing activation functions, observing unstable signals, training unusually deep unnormalized networks, or trying to understand why optimization behaves differently across model variants.

If a small network already trains reliably and reaches the required validation quality, elaborate initialization experiments may provide little practical value. The purpose of initialization is to make optimization easier to start, not to add complexity for its own sake.

A practical initialization workflow

For a custom feed-forward network, a defensible workflow is:

  1. identify the activation used after each trainable layer;
  2. use a fan-scaled initializer appropriate to that activation, such as Xavier for a suitable linear or tanh path and He for ReLU;
  3. preserve specialized initialization rules for embeddings, normalization parameters, gates, output heads, and pretrained components;
  4. run a representative batch before training and inspect activation scale across depth;
  5. run one backward pass and inspect gradient scale across depth;
  6. if signals collapse or grow rapidly, change one relevant factor at a time;
  7. confirm the choice with short training runs and validation behavior rather than relying only on initialization statistics.

This workflow turns initialization from a memorized formula into an observable engineering decision.

Conclusion

Xavier and He initialization solve the same underlying problem: random weights should begin at a scale that lets useful signals travel through a network. Xavier balances fan-in and fan-out for layers whose activation behavior fits its assumptions. He initialization uses a larger fan-in-scaled variance to compensate for the signal reduction introduced by ReLU-like activations.

The formulas are valuable because they connect layer width, activation behavior, and signal variance. Their assumptions are also simplified. Use them as principled starting points, preserve architecture-specific conventions, and verify the result by inspecting both activations and gradients before trusting a long training run.

Artificial Intelligence Machine Learning Neural Networks Optimization