Control Diffusion Generation with Classifier-Free Guidance
A text-conditioned diffusion model can understand a prompt yet still produce a sample that only loosely follows it. Increasing prompt influence can make the requested content more prominent, but pushing too hard can reduce variation or introduce visual artifacts.
Classifier-free guidance (CFG) gives diffusion systems a direct control for this trade-off. At each denoising step, the model produces one prediction with the condition and another without it. The sampler combines those predictions so the conditional signal can be strengthened.
This article builds a practical mental model for that combination, shows the arithmetic with a small example, and explains the implementation details that matter when tuning a diffusion pipeline.
Start with two predictions for the same noisy sample
A diffusion sampler repeatedly transforms a noisy state toward a cleaner sample. At a particular timestep, a conditional model receives the current noisy state (x_t), the timestep (t), and a condition (c), such as a text embedding.
Write its prediction as:
[ \epsilon_c = \epsilon_\theta(x_t, t, c). ]
For the same (x_t) and (t), the model can also make an unconditional prediction:
[ \epsilon_u = \epsilon_\theta(x_t, t, \varnothing). ]
Here, (\varnothing) represents an absent or null condition. The exact representation depends on the model and training setup.
The difference
[ \epsilon_c - \epsilon_u ]
captures the direction introduced by the condition for this denoising step. CFG scales that difference and adds it back to the unconditional prediction:
[ \epsilon_{\text{cfg}}
\epsilon_u + s(\epsilon_c-\epsilon_u), ]
where (s) is the guidance scale.
This equation is the core mental model: begin from the unconditional prediction, measure the change produced by the prompt, then control how strongly the sampler follows that change.
Some libraries use a different algebraic convention for their guidance parameter. A numeric value from one implementation therefore should not be copied into another without checking its formula.
A small numerical example
Treat the model output as a two-element vector for a simplified example.
Suppose the unconditional prediction is:
[ \epsilon_u = [0.2,\ 0.5] ]
and the conditional prediction is:
[ \epsilon_c = [0.5,\ 0.3]. ]
The conditional difference is:
[ \epsilon_c-\epsilon_u = [0.3,\ -0.2]. ]
With (s=1):
[ \epsilon_{\text{cfg}}
[0.2,\ 0.5] + 1[0.3,\ -0.2]
[0.5,\ 0.3]. ]
So this convention reproduces the conditional prediction at scale 1.
With (s=2):
[ \epsilon_{\text{cfg}}
[0.2,\ 0.5] + 2[0.3,\ -0.2]
[0.8,\ 0.1]. ]
The result moves past the ordinary conditional prediction in the same direction. CFG is therefore not just switching between conditional and unconditional generation. At scales above 1 under this convention, it extrapolates the conditional effect.
The vectors in this example are intentionally tiny. Real diffusion outputs contain many values, but the same element-wise arithmetic applies.
The model needs an unconditional path
CFG does not require a separate image classifier. That is the source of the term classifier-free.
A common training strategy randomly drops the conditioning signal for some training examples. The same denoising network then receives both conditional and unconditional cases. As a result, it can provide both predictions needed during guided sampling.
This design matters operationally. A checkpoint must support the unconditional path expected by its sampling procedure. Removing a prompt at inference time does not automatically produce a valid unconditional prediction for every conditional architecture.
The null condition is also model-specific. For text-conditioned systems it may involve an empty text representation or another designated conditioning value. Use the checkpoint’s documented conditioning procedure rather than inventing a substitute.
Guidance scale changes a trade-off, not a quality dial
It is tempting to treat a larger guidance scale as a generic quality setting. That interpretation is too simple.
Increasing the scale strengthens the difference between conditional and unconditional predictions. This can make generated samples align more strongly with the condition, but strong guidance can also narrow the range of outputs and push denoising predictions into less typical regions.
In practice, several effects can move together:
- prompt adherence can increase;
- sample diversity can decrease;
- saturation or other artifacts can become more visible;
- sensitivity to prompt wording can increase.
The useful scale range depends on the checkpoint, sampler, prediction parameterization, conditioning system, and other pipeline choices. A value that works well for one diffusion system is not a portable default.
Treat guidance scale as a parameter to evaluate for a specific pipeline and task.
CFG usually costs extra model evaluation
The straightforward form of classifier-free guidance needs both (\epsilon_c) and (\epsilon_u) at every guided denoising step.
That means the denoiser must evaluate two conditioning cases. Implementations often concatenate the conditional and unconditional inputs into one larger batch so a single batched model call produces both outputs. This can improve hardware utilization, but it does not make the extra computation disappear.
The practical cost depends on the model, accelerator, batch size, memory pressure, compiler, and serving stack. A pipeline can become limited by memory before arithmetic throughput, so doubling the effective denoiser batch can affect latency differently across systems.
When latency matters, measure the complete sampling path rather than assuming that a batched CFG implementation has the same cost as an unguided pass.
Keep the two branches identical except for conditioning
A clean CFG comparison uses the same noisy sample and timestep for both predictions. The conditioning input is the intended difference.
Conceptually:
conditional = denoiser(x_t, t, condition)
unconditional = denoiser(x_t, t, null_condition)
guided = unconditional
+ scale * (conditional - unconditional)If the branches accidentally receive different noisy states, timestep values, scheduler inputs, or model modes, their difference no longer isolates the conditioning effect.
This is especially easy to get wrong in custom batching code. After concatenating two branches, verify that tensors are split back in the same order in which they were assembled.
Also keep data types and device placement consistent. Silent casts or movement between devices can add cost and make debugging harder even when the arithmetic remains valid.
Prediction parameterization changes where guidance is applied
The notation above uses (\epsilon)-prediction because it is easy to explain, but diffusion models can use other prediction targets.
A model might predict noise, a velocity-like quantity, a denoised sample, or another parameterization defined by its training objective and scheduler. The sampler must combine outputs in the representation expected by that pipeline.
Do not take a CFG formula written for one prediction type and insert it blindly into another sampler. Follow the model and scheduler definitions together. A conversion that is mathematically valid in one formulation may need timestep-dependent factors in another.
The broader idea stays the same: compare conditional and unconditional predictions in a compatible representation, scale their difference, and pass the correctly interpreted result to the next sampling calculation.
Negative conditioning changes the reference branch
Many practical text-to-image pipelines support a negative prompt. In such systems, the branch often described casually as “unconditional” may actually use negative conditioning rather than a truly empty condition.
The same vector structure can still be used:
[ p_{\text{guided}}
p_{\text{negative}} + s(p_{\text{positive}}-p_{\text{negative}}). ]
This shifts the interpretation. The sampler is now moving away from one conditioning signal toward another, rather than moving from an empty condition toward the positive prompt.
Exact negative-prompt behavior is implementation-specific. Check the pipeline documentation before assuming that an empty negative prompt, a null condition, and omitted conditioning are identical.
Common mistakes produce misleading tuning results
Copying a scale without checking the convention
Two codebases can expose a parameter called guidance_scale while defining its arithmetic differently. Compare formulas first, then compare values.
Tuning guidance while changing several other controls
Changing the sampler, number of denoising steps, random seed policy, prompt, and guidance scale at once makes the effect of CFG difficult to isolate.
For a useful evaluation, hold the rest of the pipeline fixed and test several scales over a representative prompt set. Use multiple seeds when output diversity matters.
Evaluating only the most attractive sample
A single favorable image can hide reduced diversity or unstable behavior. Inspect repeated generations across prompts that reflect the actual application.
For a product pipeline, evaluation criteria might include condition adherence, artifact rate, diversity, latency, and memory use. The right balance depends on the product rather than one visual preference.
Assuming stronger conditioning fixes missing model capability
CFG can amplify a conditioning direction that the model already represents. It cannot reliably add concepts, spatial relations, text rendering skill, or domain coverage that the checkpoint does not model well.
If increasing guidance repeatedly produces artifacts without satisfying the request, the limiting factor may be the model, conditioning encoder, prompt representation, or data coverage rather than the scale.
Use CFG when controllability justifies the cost
Classifier-free guidance is a good fit when a diffusion checkpoint was designed for it and condition adherence matters enough to justify extra denoiser work.
It is less attractive when an unguided model already meets the task, when latency or memory is extremely constrained, or when the checkpoint uses a different conditioning mechanism. In those cases, adding CFG machinery can increase complexity without solving the actual bottleneck.
For deployment, choose the scale from evaluation data rather than from a familiar default. Test prompts that represent easy cases, difficult cases, and cases where excessive guidance could be harmful. Record the scale together with the sampler and checkpoint version because those settings form one inference configuration.
Treat guidance as controlled extrapolation
Classifier-free guidance becomes easier to reason about once it is viewed as vector arithmetic rather than a mysterious prompt-strength knob. The model gives a reference prediction and a conditioned prediction. Their difference estimates the conditioning effect for the current denoising step, and the guidance scale controls how far the sampler moves along that direction.
That mental model also exposes the main engineering checks: preserve matched inputs across both branches, respect the model’s prediction parameterization, verify the scale convention, measure the extra inference cost, and evaluate adherence together with diversity and artifacts.
When those pieces are explicit, CFG becomes a parameter you can tune deliberately instead of a number copied from another pipeline.