Distill Sequence Models with Teacher-Generated Outputs

A large text generator may produce useful outputs but still be too expensive for the latency, memory, or throughput budget of a deployment. Training a smaller model on the original dataset is the obvious baseline, but it throws away information encoded in the larger model’s behavior.

Sequence-level knowledge distillation offers another option: let a capable teacher generate target sequences, then train a smaller student to reproduce those sequences. The student learns from concrete examples of what the teacher tends to produce rather than only from the original human targets or from the teacher’s next-token probabilities.

This article builds a practical mental model for sequence-level distillation, shows how the training data changes, and explains the trade-offs that matter when deciding whether teacher-generated outputs are actually useful.

Start with the training target, not the model architecture

Suppose you have input-output pairs for a short summarization task:

input:
The payment service retried the request three times after a timeout.
The fourth attempt succeeded.

human target:
Payment succeeded after three timeout retries.

A standard supervised student is trained to assign high probability to the human target given the input.

With sequence-level distillation, a teacher first generates its own target:

teacher target:
The payment succeeded on the fourth attempt after three timeouts.

The student is then trained on a pair such as:

input -> teacher target

The training objective can still be ordinary token-level cross-entropy. What makes the method sequence-level is where the target sequence came from: the teacher generated a complete output, and that output became training data.

That distinction is easy to miss. Sequence-level distillation doesn’t require a special student architecture, and it doesn’t mean the loss must compare two whole sequences with a single scalar score.

The mental model: turn teacher behavior into data

For an autoregressive student with parameters θ, training on a teacher-generated sequence y* = (y1, ..., yT) can use the familiar negative log-likelihood objective:

L(θ) = - Σ_t log pθ(y*t | x, y*<t)

Here, x is the input and y*<t is the teacher-generated prefix before position t.

The student therefore learns a normal conditional generation problem. The unusual step happened earlier, when the teacher converted each input into a concrete target sequence.

A simple pipeline looks like this:

original inputs
     |
     v
teacher generation
     |
     v
teacher-generated dataset
     |
     v
student training

This framing has a useful engineering consequence: teacher inference and student training can be separated. You can generate the distilled dataset once, inspect it, version it, and train multiple student configurations against the same targets.

It also exposes a limitation. Once you save only the generated sequence, most of the teacher’s uncertainty is gone. If the teacher considered several continuations plausible but emitted one, the student sees that chosen sequence rather than the full distribution over alternatives.

How sequence-level and token-level distillation differ

The term knowledge distillation also describes training a student to match a teacher’s probability distribution. For a language model, token-level distillation can compare the teacher and student distributions over the vocabulary at each position.

Conceptually:

token-level distillation:
input + prefix -> teacher probabilities -> student matches probabilities

sequence-level distillation:
input -> teacher generates sequence -> student trains on generated sequence

Token probabilities contain information that a single generated target cannot show. If a teacher assigns similar probability to timeout and network timeout, a soft distribution can expose that relationship. A saved sequence containing only timeout cannot.

Sequence-level targets have a different advantage: they represent a coherent continuation selected by the teacher. The student isn’t independently told which token distributions to imitate at prefixes taken only from a reference target; it trains directly on a complete trajectory produced by the teacher.

The approaches aren’t mutually exclusive. A training system can use teacher-generated sequences while also applying token-level distillation where teacher logits are available. That combination costs more storage or teacher computation, so it should earn its complexity in evaluation rather than being treated as an automatic upgrade.

Why teacher-generated targets can be easier for a student

Many generation tasks allow several valid outputs for the same input. A human-written dataset can reflect different styles, word choices, levels of detail, and annotation conventions.

Imagine three training examples that express similar outcomes:

resolved after retry
succeeded on the second attempt
second request completed successfully

That variation isn’t inherently bad. It may represent the real task. But a small model has less capacity than its teacher and may struggle to model all valid modes equally well.

A fixed teacher often generates outputs with more consistent preferences. Its distilled targets may repeatedly use similar phrasing or structural choices for similar inputs. This can make the empirical target distribution narrower and therefore easier for a smaller student to fit.

The word easier needs a caveat. Simplifying the target distribution can help optimization while also removing useful diversity. If the application needs varied writing, broad coverage, or faithful reproduction of rare human conventions, a narrower teacher-generated dataset can be a regression even when the student’s average benchmark score improves.

Build the distilled dataset deliberately

The smallest useful implementation has three stages: select inputs, generate teacher outputs, and train the student on those generated pairs.

For each training input x_i:

y_teacher_i = generate(teacher, x_i)
save(x_i, y_teacher_i)

Then train the student as ordinary supervised generation:

for x, y_teacher in distilled_dataset:
    loss = negative_log_likelihood(student, x, y_teacher)
    update(student, loss)

This is pseudo-code, not an API prescription. Production details depend on the model family and training stack.

The generation configuration is part of the dataset definition. Greedy decoding, beam search, or stochastic sampling can produce different targets from the same teacher. If generation is stochastic, random seeds and sampling parameters affect reproducibility.

Treat those settings the way you’d treat preprocessing code. Record them alongside the teacher identifier and the source-data version. Otherwise, two datasets described as “distilled from the same teacher” may contain meaningfully different supervision.

Don’t replace the evaluation set with teacher outputs

A distilled training set answers one question: what behavior should the student learn from the teacher? Evaluation should answer a different question: does the resulting student solve the real task?

If you evaluate only against teacher-generated targets, a student can look strong because it copies teacher preferences well. That doesn’t establish that those preferences are correct or desirable.

Keep an evaluation set tied to the application objective. Depending on the task, that may mean human reference outputs, task-specific automatic metrics, factuality checks, structured validation, or human review. Compare at least three useful baselines when practical:

• the teacher;
• the same student trained on the original targets;
• the student trained with the distilled targets.

This comparison separates two effects that are otherwise easy to confuse: benefits from the student architecture or training recipe, and benefits specifically caused by distillation.

Teacher mistakes become supervision

Sequence-level distillation can faithfully transfer behavior you didn’t want.

Suppose the source document says:

The deployment was postponed from Tuesday to Friday.

If the teacher generates:

The deployment happened on Friday.

and you store that output as the target, the student receives a factual error as a positive example. Repeating this process at scale can turn systematic teacher mistakes into systematic training signals.

Filtering can reduce obvious failures. For structured tasks, validate schemas, labels, required fields, or constraints before accepting generated targets. For factual generation, automatic checks may catch some contradictions, but they shouldn’t be assumed to prove correctness.

A practical compromise is to retain original targets for cases where the teacher output fails a reliable validation rule. Another is to mix original and teacher-generated examples rather than replacing the original targets completely. The right mixture is task-dependent and should be selected using held-out evaluation.

One target per input can hide useful alternatives

A teacher defines a distribution over possible outputs, but a basic distilled dataset usually stores one sequence per input. This turns a rich distribution into a point target.

For deterministic tasks, that may be acceptable. If the output must be a normalized product category or a fixed command, diversity may offer little value.

For open-ended generation, the loss is more consequential. A writing assistant trained on one deterministic teacher output per prompt may learn a narrower style than desired.

Generating multiple teacher outputs can restore some variety:

input A -> teacher output A1
input A -> teacher output A2
input A -> teacher output A3

But more targets increase generation cost and dataset size, and stochastic samples can include weaker outputs. Multiple generations are useful only if the added diversity survives quality checks and improves the downstream evaluation you care about.

Distillation doesn’t guarantee cheaper inference

The teacher is normally used offline, so its generation cost is paid while creating the training data. The deployment savings come from the student, not from the distillation procedure itself.

A smaller student can require less memory and computation, but parameter count alone doesn’t determine serving performance. Architecture, sequence length, batching, numerical precision, hardware utilization, and decoding strategy all affect latency and throughput.

Measure the deployed configuration directly. A student that preserves quality but misses the actual latency target hasn’t solved the deployment problem.

There is also an up-front cost trade-off. Generating a large distilled corpus with an expensive teacher can consume substantial compute. If the original supervised dataset already trains a small model to the required quality, distillation may add cost without enough benefit.

Common mistakes when applying sequence-level distillation

The most common failure is treating teacher outputs as ground truth. They are synthetic labels produced by another model. Their quality is bounded by the teacher, its decoding procedure, and the inputs used to generate them.

Another mistake is changing several variables at once. If you switch student architecture, training data, optimizer settings, and decoding strategy together, you won’t know whether sequence-level distillation helped. Keep a comparable non-distilled student baseline.

Data leakage deserves the same care as in ordinary model training. Don’t generate distilled training targets from examples that belong in a held-out test set, then train the student on them. The fact that labels came from a teacher doesn’t make test inputs safe to train on.

Finally, avoid assuming that closer imitation of the teacher is the deployment objective. A smaller model can sometimes benefit from teacher behavior, but the real target is application quality under the required cost and latency constraints.

When sequence-level knowledge distillation is a good fit

Sequence-level distillation is worth testing when you already have a strong generator, need a smaller deployment model, and can generate a representative set of inputs offline. It is particularly attractive when storing full teacher probability distributions would be expensive or inconvenient but storing generated text is straightforward.

It is less compelling when the task is already solved well by direct supervised training, when teacher outputs are difficult to validate, or when output diversity is itself a core requirement. If the student can query teacher logits cheaply during training, token-level distillation may preserve information that single generated targets discard.

The choice isn’t philosophical. Build the cheapest credible baseline first, then test whether teacher-generated targets improve the quality-versus-serving-cost curve.

Treat the distilled corpus as a model artifact

The most useful way to think about sequence-level knowledge distillation is not “a smaller model copies a larger model.” It is “a teacher creates a new supervised dataset whose targets encode its generation behavior.”

That mental model makes the engineering responsibilities clearer. Inspect the synthetic targets, record how they were generated, keep evaluation independent from the teacher, and measure the student under the serving constraints that motivated compression in the first place.

If the distilled corpus improves a smaller model on those real criteria, keep it. If it merely makes the student resemble the teacher more closely, you’ve optimized the imitation step rather than the product.