Making a neural network deeper gives it more transformations to work with, but depth alone does not make optimization easy. A stack of layers must learn useful transformations while gradients travel backward through every stage. As the stack grows, that optimization path can become difficult even when the deeper model has enough capacity to represent a good solution.

Residual connections change what a block is asked to learn. Instead of making the block produce an entirely new representation, they let it learn a change to the representation it already received. The original input travels along a shortcut and is added back to the learned branch.

This small architectural idea appears in residual networks and many transformer designs. By the end of this article, you will be able to reason about the residual equation, understand why the shortcut helps optimization, handle shape changes correctly, and recognize cases where a residual connection alone is not enough to make a network stable.

Start with the residual equation

Suppose a block receives a vector x. A conventional stack of layers can be described as a function H:

y = H(x)

The block must learn the full mapping from x to y.

A residual block instead separates the output into the existing representation and a learned update:

y = x + F(x)

F(x) is called the residual branch. The path carrying x directly to the addition is the skip connection or shortcut connection.

A useful mental model is:

output = current representation + learned correction

For example, imagine that a three-dimensional representation entering a block is:

x    = [2.0, -1.0, 0.5]

The residual branch learns this update:

F(x) = [0.1,  0.3, -0.2]

The block returns:

y    = [2.1, -0.7, 0.3]

The block did not need to reconstruct the values already present in x. It only produced the change that should be added to them.

That interpretation is especially useful when the identity mapping is already a reasonable starting point. If a block does not need to change its input much, learning a residual near zero can be easier than learning a complete identity-like transformation through several nonlinear layers.

Why the shortcut changes optimization

The forward equation is simple, but the backward path explains much of the value of residual connections.

For:

y = x + F(x)

the derivative of y with respect to x contains two terms:

dy/dx = I + dF/dx

Here I is the identity term created by the shortcut. During backpropagation, the gradient reaching x therefore has a direct contribution through that identity path in addition to the contribution through F.

Without the shortcut, a deep stack depends entirely on gradients passing through the derivatives of its learned transformations. Repeated multiplication by those derivatives can make optimization difficult when their magnitudes become poorly conditioned. The residual path provides another route that does not require traversing every operation inside F.

This does not mean residual networks cannot have vanishing, exploding, or otherwise unstable gradients. The residual branch, normalization layers, activation functions, initialization, optimizer, and network depth still matter. The useful claim is narrower: an identity shortcut creates a direct gradient path that generally makes very deep architectures easier to optimize than the same stack without those shortcuts.

A residual block still learns a real transformation

It is easy to misread y = x + F(x) as if the network mostly copies its input. The addition does not prevent substantial change. If F(x) is large or structured, the output can differ greatly from x.

A typical residual branch might contain several operations:

x ------------------------------+
|                                |
+-> linear/conv -> activation ---+-> add -> y

In practice, normalization and additional layers are often included as well. The exact ordering depends on the architecture.

The important architectural distinction is between two paths:

  • the shortcut path, which preserves or adapts the existing representation;
  • the residual branch, which computes the update.

Both paths participate in the final output. The shortcut is not a backup route used only when the learned branch fails.

Addition requires compatible shapes

Element-wise addition only works when the two paths have compatible shapes. If x and F(x) both have shape [batch, 256], the operation is straightforward:

x      : [batch, 256]
F(x)   : [batch, 256]
y      : [batch, 256]

A problem appears when the block changes representation size. Suppose the residual branch produces 512 features:

x      : [batch, 256]
F(x)   : [batch, 512]

You cannot directly add those tensors. One common solution is a learned projection P on the shortcut:

y = P(x) + F(x)

with:

P(x)   : [batch, 512]
F(x)   : [batch, 512]

Convolutional residual networks similarly use projections when changing channel count or spatial resolution. A projection solves the shape mismatch, but it changes an important property: the shortcut is no longer a pure identity mapping. Gradients through that path now also depend on the projection.

Use a projection when the architecture requires a shape change, not merely because every shortcut seems as though it should contain trainable parameters. When shapes already match, an identity shortcut is simpler and preserves the direct path.

Addition and concatenation are different operations

Skip connections are sometimes confused with concatenating an earlier representation into a later one.

Residual addition does this:

x + F(x)

If both inputs have 256 features, the result still has 256 features.

Concatenation does this conceptually:

concat(x, F(x))

If both inputs have 256 features, the result has 512 features along the concatenated dimension.

These operations create different architectures. Addition forces both paths to contribute in the same representation space. Concatenation preserves both representations separately and leaves a later operation to mix them. Both can be useful, but concatenation is not a drop-in substitute for a residual connection because it changes dimensions, parameter requirements, and downstream computation.

Residual placement matters in transformers

Transformers use residual connections around major sublayers such as attention and feed-forward networks. A simplified block can be written in different ways depending on where normalization occurs.

A post-normalization pattern can be sketched as:

h1 = norm(x + attention(x))
y  = norm(h1 + feed_forward(h1))

A pre-normalization pattern moves normalization before each residual branch:

h1 = x  + attention(norm(x))
y  = h1 + feed_forward(norm(h1))

These equations are simplified; real transformer implementations may include dropout, biases, gating, or other details. The key point is that saying a transformer “has residual connections” does not fully specify its optimization behavior. The placement of normalization relative to the shortcut changes the computation and the gradient path.

When implementing or reproducing a model, follow the architecture’s stated ordering rather than moving normalization across the residual addition because the two forms look similar.

Watch the scale of the residual branch

Residual addition assumes the two paths can be combined meaningfully, but their magnitudes do not have to be equal. Problems can appear when residual updates become disproportionately large as many blocks are stacked.

Consider a simplified sequence:

x1 = x0 + F1(x0)
x2 = x1 + F2(x1)
x3 = x2 + F3(x2)

Every block contributes another update. If those updates have poorly controlled scale, activation magnitudes or gradients can become difficult to manage in a very deep network.

Architectures address this in different ways, including normalization, careful initialization, explicit residual scaling, or designs that initialize parts of the residual branch near zero. These techniques are not interchangeable guarantees. Their usefulness depends on the architecture and training setup.

The practical lesson is to inspect both paths when debugging instability. If activations after residual additions grow rapidly with depth, changing only the optimizer may treat the symptom rather than the architectural cause.

Common mistakes when implementing residual blocks

Applying the shortcut to the wrong tensor

The value added at the end of a block should be the intended shortcut input, possibly transformed by a documented projection. Accidentally overwriting that tensor before saving the shortcut changes the function being implemented.

A clear pseudo-code pattern is:

shortcut = x
residual = transform(x)
y = shortcut + residual

If a projection is required:

shortcut = project(x)
residual = transform(x)
y = shortcut + residual

Keeping the two paths explicit makes shape and ordering bugs easier to see.

Assuming a projection is an identity shortcut

A learned projection can make dimensions compatible, but it is not mathematically the same as passing x unchanged. This distinction matters when reasoning about gradient paths and when comparing architectures.

Adding tensors that mean different things

Matching dimensions are necessary but not sufficient. Two tensors can have the same shape while representing incompatible quantities. Residual addition works best when both paths live in a compatible feature space by design.

Treating residual connections as a complete stability solution

Residual connections improve the optimization structure, but they do not choose a suitable learning rate, prevent bad data, fix an incorrect loss, or guarantee stable activation scales. If training diverges, inspect the residual architecture alongside initialization, normalization, precision, optimizer settings, and data.

When residual connections are a good fit

Residual connections are particularly useful when building deep stacks in which successive blocks refine representations while keeping the same basic feature structure. This includes many convolutional networks and transformer-style architectures.

They are less natural when the two paths do not represent compatible quantities or when an operation intentionally changes the representation in a way that cannot sensibly be added back to the input. In those cases, a projection may make addition possible, but making tensor shapes match does not automatically make the architecture conceptually sound.

For a small or shallow network that already trains reliably, adding residual structure may provide little practical benefit and can make the model more complex to reason about. Architecture changes should solve an observed optimization or representation problem rather than be added mechanically.

Conclusion

A residual connection turns a block from “produce the whole next representation” into “produce an update to the current representation.” The basic equation is only y = x + F(x), but the shortcut changes both the forward computation and the path available to gradients.

When using residual connections, keep three checks in mind: make the shortcut and residual shapes compatible, preserve the intended ordering of normalization and other operations, and monitor the scale of repeated residual updates. With those details correct, residual structure provides a reusable way to make deep neural networks easier to optimize without requiring every block to rebuild useful information from scratch.

Artificial Intelligence Machine Learning Neural Networks Model Architecture