A large model may produce useful predictions but still be too expensive or slow for the environment where it must run. A mobile application, an edge device, or a high-volume service can have tighter limits on memory, latency, and compute.

Knowledge distillation is one way to address that gap. Instead of training a smaller model only from the original labels, we also train it to imitate information produced by a stronger teacher model. The smaller model is called the student.

The goal is not to copy the teacher perfectly. It is to transfer useful behavior into a model that is cheaper to run while retaining enough quality for the intended task.

The core idea: learn from more than the correct label

Consider a three-class image classifier. For one training image, the label says only that the correct class is cat:

hard label
cat: 1
 dog: 0
 fox: 0

That label is necessary, but it contains little information about relationships among the alternatives. A trained teacher might instead produce probabilities such as:

teacher probabilities
cat: 0.72
 dog: 0.23
 fox: 0.05

The distribution says more than “cat is correct.” It also indicates that the teacher considers dog more plausible than fox for this example.

Distillation gives the student access to this richer signal. During training, the student can learn both from the ground-truth label and from the teacher’s output distribution.

These teacher outputs are often called soft targets because probability is distributed across multiple classes rather than concentrated entirely on one label.

A typical distillation objective

For a classification task, a simple distillation loss combines two objectives:

total loss = label loss + distillation loss

More explicitly, we can write:

L = alpha * L_hard + (1 - alpha) * L_soft

L_hard measures the student’s error against the real labels. L_soft measures how different the student’s predictions are from the teacher’s predictions. alpha controls the balance between those signals.

The exact loss depends on the task and implementation. For multiclass classification, the hard-label term is commonly cross-entropy, while the teacher-student term commonly compares probability distributions using cross-entropy or KL divergence.

Keeping the original labels in the objective is important. A teacher can be wrong. Training only to reproduce teacher outputs can transfer teacher errors as faithfully as teacher strengths.

Temperature reveals relationships between alternatives

A confident classifier can assign nearly all probability to one class. That makes the remaining probabilities very small and reduces the information available in their relative values.

Distillation commonly uses a temperature parameter when converting logits into probabilities. Given logit z_i for class i, the softened probability is:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

With T = 1, this is the ordinary softmax. A temperature greater than 1 produces a flatter distribution.

Suppose a teacher has logits that strongly favor cat. Raising the temperature can make the lower-ranked classes easier to compare:

low temperature:
cat  0.97
 dog  0.02
 fox  0.01

higher temperature:
cat  0.65
 dog  0.22
 fox  0.13

The second distribution exposes more of the teacher’s view of the alternatives. The precise probabilities depend on the logits and chosen temperature; the values above only illustrate the effect.

When temperature is used for the teacher targets, the student distribution used in the distillation term should be computed with the same temperature. A common formulation also scales that loss by T^2 so changing temperature does not unintentionally shrink its gradient contribution relative to the hard-label loss.

Temperature is therefore not simply a knob for making predictions less confident. In distillation, it changes the training signal and must be considered together with the loss weighting.

Distillation does not automatically make a model smaller

This distinction is easy to miss: distillation is a training method, not a compression operation that removes parameters from an existing network.

The student is smaller because we choose a smaller student architecture. Distillation then helps train that architecture using information from the teacher.

For example:

large teacher
     |
     | predictions or internal representations
     v
small student architecture
     |
     v
trained small model

If teacher and student have the same architecture and size, distillation may still be useful in some training setups, but it does not create the memory and compute reductions normally associated with model compression.

This also means student capacity matters. A very small student may simply lack enough representational capacity to reproduce the teacher’s useful behavior. Better distillation cannot guarantee that an undersized model will match a much larger one.

Decide what knowledge the student should imitate

Output probabilities are the simplest teaching signal, but they are not the only option.

Response-based distillation asks the student to imitate the teacher’s final outputs, such as class probabilities or logits. It is comparatively simple because teacher and student do not need matching internal architectures.

Feature-based distillation also aligns intermediate representations. This can provide a richer signal, but layers may have different dimensions or meanings, so additional projection or alignment mechanisms can be necessary.

Relation-based approaches transfer relationships among examples or internal features rather than requiring direct equality between individual representations.

The more internal signals a method uses, the more assumptions it tends to make about the teacher and student. Start with output-level distillation unless there is evidence that a more complicated objective is needed.

Teacher data matters as much as teacher quality

A strong teacher is useful only on examples that represent the student’s intended workload.

Imagine distilling a support-ticket classifier. If the distillation set contains mostly billing questions but production traffic includes many account-access and cancellation requests, the student receives little teacher guidance for those missing regions.

A practical dataset should cover:

  • common production cases;
  • difficult or ambiguous examples;
  • important minority classes;
  • realistic input lengths and formats;
  • cases where teacher mistakes would be costly.

Unlabeled data can also be useful because the teacher can generate targets for it. However, teacher-generated targets are not new ground truth. Systematic teacher errors can propagate into the student, especially when generated targets dominate the training objective.

Evaluate the student independently

A low distillation loss only shows that the student is becoming similar to the teacher under the chosen objective. It does not prove that the student solves the real task well.

Evaluate the final student against held-out ground truth using metrics appropriate for the application. For a classifier, that might include accuracy, precision, recall, F1, calibration, or per-class results. For other model types, different task-specific metrics are needed.

Also measure the reason the student exists in the first place:

quality
latency
throughput
memory use
model size
hardware cost

A student that loses a small amount of benchmark quality may still be a good engineering trade-off if it substantially reduces latency or cost. Conversely, a dramatically smaller model is not useful if it falls below the quality required by the product.

Compare at least three useful baselines when practical:

teacher
student trained normally
student trained with distillation

The second baseline is especially important. Without it, an improvement might be attributed to distillation even when the student architecture would have achieved similar quality through ordinary supervised training.

Watch for teacher mistakes and confidence problems

Distillation transfers behavior, not truth.

If a teacher consistently confuses two classes, its soft targets can encourage the student to learn the same confusion. A poorly calibrated teacher can also produce confidence values that should not be interpreted as reliable probabilities of correctness.

Ground-truth labels, representative validation data, and error analysis remain necessary. Examine cases where teacher and labels disagree rather than assuming the larger model must be right.

The teacher may also rely on patterns unavailable to the student. A large model can represent distinctions that a much smaller architecture cannot. In that situation, forcing the student to match every teacher output can compete with learning the primary task. Loss weights, student capacity, and validation results should guide the balance.

Distillation has a training cost

A smaller deployed model does not imply cheaper training.

Teacher inference must be performed on the distillation data. If the teacher is expensive, repeatedly evaluating it during student training can add substantial compute and latency to the training pipeline.

When the training data is fixed, one practical option is to compute and store teacher outputs once, then reuse them across student training runs. This trades storage for avoiding repeated teacher inference. It works well for output-based targets, but it is less flexible if data augmentation changes examples dynamically or the training method needs teacher activations generated on the fly.

Deployment savings should therefore be evaluated against the one-time or recurring cost of producing the student.

Know when distillation is the wrong tool

Knowledge distillation is most compelling when a strong model already exists and deployment constraints justify a smaller student.

A simpler approach may be better when:

  • the original model already meets latency and cost targets;
  • a smaller model trained normally reaches the required quality;
  • there is not enough representative data for useful teacher supervision;
  • teacher inference is too expensive relative to expected deployment savings;
  • the student’s capacity is too limited for the required task.

Distillation can also be combined with techniques such as quantization or pruning, but they solve different problems. Distillation changes how the student is trained. Quantization changes numerical representation, while pruning removes selected parameters or structures. Combining techniques can produce additional savings, but each introduces its own quality and implementation trade-offs.

A practical workflow

A disciplined distillation project can stay simple:

  1. Define the deployment constraint: latency, memory, throughput, or cost.
  2. Establish teacher quality on a held-out evaluation set.
  3. Choose a student architecture that can realistically meet the deployment target.
  4. Train that student normally to establish a baseline.
  5. Add teacher targets and tune the hard-label versus distillation balance.
  6. Evaluate task quality and deployment metrics together.
  7. Analyze teacher-student disagreements and important failure cases.
  8. Keep the distilled model only if the measured trade-off is worthwhile.

This workflow prevents a common mistake: treating teacher imitation as the objective. The real objective is a student model that satisfies application requirements more effectively than the available alternatives.

Conclusion

Knowledge distillation trains a student with information from a stronger teacher, often by combining ordinary labels with softened teacher predictions. The extra signal can help a smaller architecture retain more useful behavior than it learns from hard labels alone.

The technique is not automatic compression and it does not guarantee teacher-level quality. Its value depends on student capacity, representative data, a suitable loss, careful temperature handling, and independent evaluation against real task requirements.

Use distillation when the deployment benefit of a smaller model is concrete, then judge success by both model quality and the resource savings that motivated the student in the first place.

Artificial Intelligence Machine Learning Model Training Model Optimization