Training one neural network to solve several tasks can reduce duplicated computation and let related tasks share useful representations. It also creates a problem that single-task training does not have: two losses can ask the same shared parameter to move in opposing directions during the same update.
Simply adding the losses does not make that disagreement disappear. Their gradients are added too, so one task can partially cancel another or dominate the shared update. Gradient surgery is a family of techniques that changes task gradients before combining them. A well-known example is projected conflicting gradients, commonly called PCGrad, which removes a conflicting component of one task’s gradient relative to another.
This article develops the idea from vector geometry rather than treating it as an optimizer trick. You will learn how to detect gradient conflict, what projection changes, why conflict is not automatically harmful, and how to decide whether gradient surgery addresses the real bottleneck in a multi-task model.
Shared parameters create a shared negotiation
Consider a model with a shared encoder and two task-specific heads:
input
|
shared encoder
|-----------|
| |
head A head B
| |
loss A loss BSuppose task A classifies the topic of a support message while task B predicts its urgency. Both heads depend on the same encoder parameters theta.
A common training objective is a weighted sum:
L = w_A * L_A + w_B * L_BDifferentiation is linear, so the gradient on the shared parameters is
g = w_A * g_A + w_B * g_Bwhere
g_A = grad_theta L_A
g_B = grad_theta L_BThe optimizer ultimately sees the combined gradient g. If the task gradients point in similar directions, the tasks broadly agree about how the shared representation should change. If they point in opposing directions, their contributions interfere.
This distinction applies only where parameters are shared. Parameters belonging exclusively to head A do not receive a gradient from loss B, so there is no cross-task conflict to resolve there.
Use the dot product to identify directional conflict
For two nonzero task gradients, their dot product tells us whether their directions form an acute, right, or obtuse angle:
g_A dot g_B > 0 broadly aligned
g_A dot g_B = 0 orthogonal
g_A dot g_B < 0 conflicting directionsThe last case is the one gradient projection methods focus on. To see why, consider a tiny two-dimensional example:
g_A = [2, 1]
g_B = [-1, 1]Their dot product is
(2 * -1) + (1 * 1) = -1so the angle between them is greater than 90 degrees. If we add them directly with equal weights, we get
g_A + g_B = [1, 2]That update is not necessarily wrong. It is the gradient of the summed objective. But it hides the fact that part of task A’s preferred direction opposes task B.
A useful diagnostic is cosine similarity:
cos(g_A, g_B) = (g_A dot g_B) / (||g_A|| * ||g_B||)Cosine similarity separates direction from magnitude. A negative value indicates directional conflict, while a value near -1 indicates nearly opposite directions. Gradient norms should be inspected separately because a small gradient and a huge gradient can have the same cosine relationship while contributing very differently to the actual update.
Projection removes one conflicting component
PCGrad’s core operation is ordinary vector projection. When g_A dot g_B < 0, project g_A onto the plane perpendicular to g_B:
g_A' = g_A - ((g_A dot g_B) / ||g_B||^2) * g_BUsing the earlier example,
g_A = [2, 1]
g_B = [-1, 1]
g_A dot g_B = -1
||g_B||^2 = 2therefore
g_A' = [2, 1] - (-1 / 2) * [-1, 1]
= [2, 1] - [0.5, -0.5]
= [1.5, 1.5]Check the result:
g_A' dot g_B
= (1.5 * -1) + (1.5 * 1)
= 0The modified task-A gradient no longer contains a component pointing against task B. Its component perpendicular to g_B remains.
This is the key mental model:
original task gradient
= non-conflicting component
+ conflicting component
projection removes the conflicting componentThe operation does not discover a universally correct direction. It imposes a particular rule for handling local disagreement between objectives.
Extending the idea beyond two tasks
With two tasks, the geometry is easy to draw. With many tasks, each task gradient may conflict with several others.
A PCGrad-style procedure can be expressed conceptually as:
for each task i:
g_i_modified = g_i
visit the other tasks in a randomized order
for each task j:
if dot(g_i_modified, g_j) < 0:
g_i_modified = g_i_modified
- dot(g_i_modified, g_j) / ||g_j||^2 * g_j
combine the modified task gradientsThe randomized order matters because sequential projections are generally order-dependent. Projecting against task B changes the vector that may later be compared with task C. Reversing those operations can produce a different result.
That means gradient surgery should not be described as computing a unique conflict-free optimum. It is an update rule with its own choices and stochasticity.
Implementations also need to define what happens when a reference gradient has zero or near-zero norm. The projection formula divides by ||g_j||^2; a zero gradient provides no meaningful direction to project against. Practical code should skip such a projection or otherwise handle the numerical boundary explicitly.
Measure the problem before changing the update rule
A training curve that improves slowly is not evidence of gradient conflict by itself. Before adding gradient surgery, collect task-level information.
For a manageable number of tasks, useful measurements include:
per-task loss
per-task validation metric
per-task shared-gradient norm
pairwise gradient cosine similarityThe pairwise cosine values can be summarized over many batches. For example, track the fraction of measured task pairs whose cosine similarity is negative, together with the distribution of gradient norms.
Suppose a two-task model shows this pattern:
early training late training
negative cosine 18% 62%
||g_A|| median 0.8 0.3
||g_B|| median 0.7 4.2The late-training issue is not merely that conflicts occur more often. Task B has also become much larger in gradient magnitude. Loss scaling, task sampling, label quality, or different convergence rates may be part of the problem.
This is why a conflict metric is a diagnostic rather than a verdict. A negative dot product says two local directions disagree. It does not say which task deserves priority or whether removing the disagreement improves the metrics you care about.
Loss weighting and gradient surgery solve different problems
It is easy to treat all multi-task optimization techniques as interchangeable, but they act on different aspects of the update.
Loss weighting changes how strongly each task contributes:
L = 0.2 * L_A + 1.0 * L_BThis scales the corresponding task gradients before they are combined. It can address an intentional priority difference or a severe magnitude imbalance, but scaling a gradient by a positive number does not change its direction. Two opposing gradients remain opposing.
Gradient projection changes direction when a conflict condition is met. It does not by itself encode business priority, guarantee equal task progress, or correct poorly calibrated loss scales.
These methods can therefore be complementary, but combining them requires a clear definition of the intended objective. If task A is deliberately five times more important than task B, an update rule that treats all conflicts symmetrically may work against that policy unless the weighting and projection procedure are designed consistently.
Conflict can be real without being a bug
Multi-objective learning contains genuine trade-offs. Imagine a shared model where one task rewards invariance to a feature while another task needs that same feature to make predictions. Their gradients may conflict because the tasks want incompatible representations, not because optimization is malfunctioning.
Projection cannot create model capacity that does not exist. If a single shared representation is structurally unsuitable for both tasks, architectural changes may be more appropriate:
more shared layers -> stronger sharing, more opportunity for interference
fewer shared layers -> more task specialization
separate adapters/heads -> targeted task-specific capacity
separate models -> no shared-parameter gradient conflictThe last option costs more memory or inference compute in many deployments, but it may be simpler when tasks are weakly related or have different serving requirements.
A second source of apparent conflict is noisy mini-batch estimation. The gradients you measure are sample estimates. Two tasks can look opposed on one batch and aligned on another. Decisions should therefore be based on repeated measurements and validation behavior rather than a single cosine value.
Account for the implementation cost
Ordinary training can form one scalar combined loss and perform one backward pass. Measuring or modifying separate task gradients usually requires retaining or recomputing enough information to obtain those gradients individually.
The exact memory and compute cost depends on the framework, model architecture, number of tasks, and implementation strategy. With many tasks, pairwise comparisons also grow quadratically in the number of tasks:
T tasks -> T * (T - 1) / 2 unordered pairsFor T = 4, that is 6 pairs. For T = 20, it is 190.
This does not mean gradient surgery is impractical. It means the optimization benefit should be compared with its training-system cost. If tasks already train well together, extra backward computation and gradient bookkeeping may solve a problem that is not operationally important.
When only a subset of parameters is shared, measuring conflict on those shared parameters can also be more informative and cheaper than flattening every gradient in the model.
Avoid common interpretation mistakes
Treating every negative cosine as harmful. Stochastic optimization naturally produces variable gradients. What matters is persistent conflict together with undesirable task-level outcomes.
Ignoring gradient magnitude. Cosine similarity describes angle, not influence. A tiny opposing gradient may have little effect on the summed update, while a large one can dominate it.
Projecting task-specific parameters. Conflict is meaningful only where multiple tasks act on the same parameters. Separate heads should normally receive their own task gradients.
Assuming projection preserves the original summed objective. Once task gradients are modified before combination, the resulting direction generally differs from the gradient of the original weighted sum of losses.
Using training loss as the only judge. Multi-task systems exist to serve task-level goals. Compare validation metrics for every important task and include operational metrics such as training cost when relevant.
Expecting optimization to repair task incompatibility. Persistent conflict may indicate that tasks should share less of the network or should not be trained as one model at all.
A practical evaluation workflow
When a multi-task model underperforms, start with the simplest baseline: a weighted sum of task losses and a standard optimizer. Record task-level validation metrics rather than only the total training loss.
Next, measure shared-gradient norms and pairwise cosine similarities on a representative sample of training steps. Look for repeated relationships: does one task dominate in magnitude, do particular pairs conflict persistently, and does conflict coincide with stalled or degrading validation performance?
If magnitude imbalance is the main pattern, investigate loss definitions, weights, and task sampling first. If directional conflict is persistent and appears connected to poor joint training, test gradient projection as a controlled experiment.
Keep the comparison fair. Use the same data split, model architecture, evaluation procedure, and comparable training budget. Report each task separately. A method that improves the average metric by sacrificing a critical task may be unacceptable even if the aggregate number rises.
Finally, compare against a simpler architectural baseline when practical. Giving tasks more private capacity can sometimes address interference more directly than adding a sophisticated update rule.
When gradient surgery is a good fit
Gradient projection is most worth investigating when several objectives genuinely share parameters, task-level gradients show persistent directional conflict, and the joint model performs worse than its capacity or single-task baselines suggest it should.
It is less compelling when tasks already cooperate, when one task is explicitly subordinate to another, when most parameters are task-specific, or when the main problem is data quality, label noise, sampling imbalance, or insufficient model capacity. In those cases, changing gradient geometry can distract from the actual constraint.
The reusable lesson is broader than PCGrad: in multi-task learning, a combined loss hides a negotiation among task gradients. Inspect both their magnitudes and directions before deciding how to intervene. Projection is one defensible way to handle local directional conflict, but its value comes from matching a measured optimization problem rather than from applying it by default.