A model can be accurate enough for a product and still be too expensive to deploy. A large classifier may exceed a mobile memory budget, miss a latency target, or cost too much when every request requires substantial compute. Replacing it with a smaller model reduces those costs, but training the smaller model only from ground-truth labels can leave useful information behind.

Knowledge distillation addresses this problem by training a smaller student model to learn from a stronger teacher model. Instead of seeing only the correct class, the student can also learn how the teacher distributes its confidence across the alternatives.

This article builds the idea from a simple classification example. You will learn what the teacher actually transfers, why temperature is useful, how the distillation loss fits together, what can go wrong, and when ordinary supervised training is the simpler choice.

Start with information that hard labels discard

Suppose an image belongs to the class cat. Its ground-truth target is effectively:

cat: 1
fox: 0
dog: 0
car: 0

That target tells a model which answer is correct, but nothing about relationships among the wrong answers.

Now imagine a strong teacher produces these probabilities for the same image:

cat: 0.82
fox: 0.03
dog: 0.14
car: 0.01

The teacher still favors cat, but its output contains more structure. It considers dog substantially more plausible than car. A student trained to reproduce that distribution receives a richer target than it gets from the one-hot label alone.

This is the central mental model for distillation:

hard target: learn which class is correct
soft target: also learn how the teacher relates the alternatives

The teacher’s probabilities are not ground truth. They encode the teacher’s learned behavior, including its useful distinctions and its mistakes. Distillation is therefore imitation guided by a model, not a way to manufacture additional truth.

Temperature exposes more of the teacher’s distribution

A classifier usually produces raw scores called logits before converting them to probabilities with softmax. For class i, a temperature-scaled softmax is:

p_i(T) = exp(z_i / T) / sum_j exp(z_j / T)

where z_i is the logit and T is the temperature.

At T = 1, this is the usual softmax. For T > 1, dividing the logits by a larger number reduces their differences, so the resulting probability distribution becomes softer.

Consider teacher logits:

cat: 4
dog: 2
fox: 1

At the normal temperature, the highest logit dominates. Raising the temperature makes the lower-ranked classes receive more probability mass while preserving the ordering of finite logits. The student can then see more clearly that the teacher considers dog closer to cat than fox.

This matters because a highly confident teacher may otherwise produce probabilities so close to one-hot targets that little additional information remains in the non-target classes.

Temperature does not improve the teacher or make its probabilities better calibrated. In distillation it is a training mechanism for exposing relative logit structure. The student normally returns to the ordinary inference setup after training unless the application independently requires a different output transformation.

Train the student on two objectives

A practical classifier distillation setup commonly combines two signals:

  1. the real labels from the dataset;
  2. the teacher’s softened output distribution.

Let the teacher and student logits be z_t and z_s. Using the same distillation temperature T, define:

q_teacher = softmax(z_t / T)
q_student = softmax(z_s / T)

The distillation term encourages the student distribution to match the teacher distribution. Cross-entropy or KL divergence can express this matching objective, depending on the implementation.

The student also keeps an ordinary supervised loss against the real label at the normal temperature. Conceptually:

L_total = alpha * L_label + beta * L_distill

alpha and beta control how strongly training trusts the dataset labels versus teacher behavior.

When using a high temperature, implementations often scale the distillation term by T^2. The reason is not that T^2 changes the desired teacher distribution. Increasing temperature changes the magnitude of gradients flowing through the softmax; the scaling compensates for that effect so changing T does not unintentionally shrink the distillation signal as severely.

Exact loss conventions differ across implementations. Some expose weights that already account for temperature scaling, and some define KL-divergence arguments differently. Treat the mathematical objective as the source of truth rather than copying a loss expression without checking the library’s reduction and weighting semantics.

A minimal training loop

The essential procedure does not require a special model architecture. The teacher and student only need compatible outputs for the quantity being matched.

For a classifier, the training loop can be expressed as pseudocode:

freeze teacher

for inputs, labels in training_data:
    with no_gradient:
        teacher_logits = teacher(inputs)

    student_logits = student(inputs)

    label_loss = cross_entropy(student_logits, labels)

    teacher_soft = softmax(teacher_logits / T)
    student_log_soft = log_softmax(student_logits / T)
    distill_loss = kl_divergence(teacher_soft, student_log_soft)

    loss = alpha * label_loss + beta * T^2 * distill_loss

    update student using loss

This is intentionally framework-neutral pseudocode. A concrete library may expect KL-divergence arguments in another order or may apply a particular reduction. Those details must be checked before translating the example directly into code.

The important data flow is simpler than the API details:

input ---------> teacher ---------> soft target
  |
  +------------> student ---------> student output
                    |
real label --------+----> combined training objective

The teacher is used to generate a target. Only the student is optimized.

Distillation changes training cost, not just inference cost

The reason to distill is usually a deployment constraint: model size, memory, latency, throughput, energy use, or serving cost. But the technique introduces extra work during training.

If teacher outputs are computed online, every student training batch also requires a teacher forward pass. A large teacher can therefore make student training substantially more expensive than ordinary supervised training.

For a fixed transfer dataset, one alternative is to precompute teacher logits or probabilities. That removes repeated teacher inference from later student epochs, but creates a storage and data-management cost. Precomputation also becomes awkward when training uses transformations that change the teacher input dynamically.

The relevant trade-off is therefore lifecycle-wide:

extra training compute and complexity
                versus
lower repeated inference cost after deployment

Distillation is most attractive when the deployed student will run often enough, or under tight enough constraints, for the inference savings to justify the additional training pipeline.

Student capacity still sets a limit

A teacher can provide a better learning signal, but it cannot make a student architecture arbitrarily expressive.

Imagine a teacher that separates classes using complex features while the student has very little capacity. The student may understand from the soft targets which examples the teacher considers similar, yet still lack enough parameters or suitable structure to reproduce those distinctions.

This creates an important design rule: choose the student for the deployment budget first, then evaluate whether distillation improves that feasible architecture. Do not assume that a sufficiently strong teacher can compress into any desired size without loss.

A useful experiment compares at least these models on the same validation set:

teacher
student trained only on labels
student trained with distillation

Measure the task metric together with the deployment metrics that motivated compression. A distilled student that gains a small amount of accuracy but still misses the latency or memory target has not solved the original problem.

The transfer data determines what the student can imitate

The teacher teaches through the examples it sees during distillation. If the transfer data does not cover an important region of the production distribution, the student gets little opportunity to observe the teacher’s behavior there.

Suppose a support-ticket classifier serves both billing and account-security requests, but the distillation dataset contains almost no security examples. Even an excellent teacher cannot communicate detailed security behavior through inputs that are absent.

Unlabeled examples can be useful because the teacher can generate targets for them, but teacher-generated targets do not remove data-quality concerns. The inputs should still represent the situations in which the student is expected to operate.

Evaluate important slices separately when mistakes have different consequences. Aggregate accuracy can hide a student that imitates common teacher behavior well while degrading on rare but important cases.

Distillation can copy teacher errors

The soft target is informative precisely because it reflects the teacher. That is also the main failure mode.

If the teacher systematically confuses two classes, assigns misleading confidence on an underrepresented group, or relies on a spurious feature, a student trained strongly toward teacher outputs can inherit that behavior.

Keeping a supervised label loss provides an independent signal, but it does not guarantee that teacher errors disappear. The relative loss weights matter, as do data coverage and student capacity.

Before distilling, evaluate the teacher on the same task and slices that will be used to judge the student. A larger model is not automatically a suitable teacher. It should demonstrate behavior worth transferring.

Do not confuse distillation with probability calibration

Temperature appears in both knowledge distillation and classifier calibration, but the goals are different.

In distillation, a temperature above one is typically used during training to soften teacher and student distributions and expose relationships among logits.

In temperature-based calibration, a temperature is fitted after model training to make reported confidence better match observed outcomes on held-out data.

The shared mathematical operation does not make the procedures interchangeable. Distillation temperature is part of a teacher-student learning objective; calibration temperature is part of adjusting predictive probabilities for decision use.

Tune against the actual deployment objective

There is no universal temperature, student size, or label-to-distillation loss ratio that is correct for every task. Treat them as experimental choices.

A practical sequence is:

  1. Train the candidate student normally to establish a baseline.
  2. Evaluate the teacher and verify that its behavior is worth transferring.
  3. Distill into the same student architecture.
  4. Compare validation quality, important slices, latency, memory, and throughput.
  5. Tune temperature and loss weighting only when the baseline experiment shows that distillation is promising.

This ordering prevents a common mistake: spending substantial time tuning distillation when the student already meets the product requirement without it.

For classification, also inspect disagreement cases between teacher and student. They often reveal whether the remaining gap comes from insufficient student capacity, weak transfer-data coverage, or teacher behavior that should not be copied.

Know when a simpler approach is enough

Knowledge distillation is useful when a strong teacher exists and a smaller deployment model needs help recovering quality. It is particularly natural when the output distribution contains meaningful structure that hard labels omit.

It is less compelling when the ordinary student already meets the required quality and resource targets. In that case, teacher inference, extra hyperparameters, and a more complicated training pipeline may add cost without solving a real problem.

Distillation is also not the same as changing numeric precision, pruning parameters, or designing a smaller architecture. Those techniques alter different parts of the deployment problem and can sometimes be combined with distillation. Which approach matters most depends on whether the bottleneck is model quality, memory, compute, hardware support, or latency.

Conclusion

Knowledge distillation treats a trained model’s output behavior as a learning signal. A teacher supplies more than the winning class: its softened distribution can show the student how alternatives relate, while the real labels keep training connected to observed truth.

The practical question is not whether distillation can make a student resemble its teacher. It is whether that resemblance produces a student that meets a concrete deployment budget without unacceptable quality loss. Start with a normally trained student, measure the gap, and add distillation when the teacher’s extra signal is worth the additional training complexity.

Artificial Intelligence Machine Learning Neural Networks Knowledge Distillation