An autoregressive model generates a sequence one element at a time. A language model predicts the next token from the tokens before it; a sequence model might similarly predict the next symbol, event, or value from an existing prefix.
That creates a practical training question: when teaching the model to predict step 5, should the input contain the correct steps 1–4 from the dataset, or the model’s own earlier predictions?
A common training method called teacher forcing uses the correct previous elements from the training sequence. This makes training efficient and gives every prediction a clean target context. It also creates a difference between training and generation: at inference time, the correct future sequence is unavailable, so the model must continue from its own outputs.
This article builds a mental model for that difference, explains what teacher forcing does and does not guarantee, and shows how to reason about errors that compound during autoregressive generation.
Start with one shifted sequence
Suppose a training example is the token sequence:
<BOS> cats sleep here <EOS><BOS> and <EOS> are simplified markers for the beginning and end of the sequence.
For next-token training, the sequence can be viewed as aligned input-target pairs:
input prefix target
<BOS> cats
<BOS> cats sleep
<BOS> cats sleep here
<BOS> cats sleep here <EOS>At each position, the model is asked to predict the next token while receiving the correct earlier tokens. That is teacher forcing.
In an implementation, the same idea is commonly expressed by shifting one token sequence:
model input: <BOS> cats sleep here
target: cats sleep here <EOS>A causal sequence model prevents each position from seeing later tokens, so one forward pass can compute predictions for all four targets. The exact tensor layout depends on the model and training library, but the learning problem is the same: predict each next token from the ground-truth prefix available before it.
Why teacher forcing is useful
The simplest alternative would be to generate a prediction, feed that prediction back into the model, generate the next one, and repeat through the entire training sequence.
Teacher forcing avoids making every later training position depend on sampled earlier outputs. This has two important consequences.
First, a bad early prediction does not replace the known training context for every later target. If the model would incorrectly predict dogs instead of cats, it can still learn the separate mapping:
<BOS> cats -> sleeprather than being forced to learn the next target from an accidental prefix it produced while still poorly trained.
Second, architectures that support parallel computation across sequence positions can evaluate many next-token predictions in one training forward pass. A causal Transformer, for example, can process the shifted training sequence while an attention mask prevents a position from using future tokens. Teacher forcing is the target-conditioning scheme; parallel execution is an architectural and implementation property, not the definition of teacher forcing itself.
This distinction matters because recurrent sequence models can also use teacher forcing even though their recurrent computation is normally processed sequentially through time.
Training and generation use different prefixes
During training, the prefix comes from the dataset:
<BOS> cats sleep -> predict "here"During free-running generation, the model has only the prompt and whatever it has generated so far:
prompt -> model token -> model token -> model token -> ...Imagine the intended continuation is:
cats sleep herebut the model generates:
cats playIts next prediction is now conditioned on cats play, not on the training prefix cats sleep. The model may have seen examples beginning with cats play during training, but it is no longer following the particular reference sequence used for this example.
This difference is often discussed under the term exposure bias: during teacher-forced training, the model is conditioned on prefixes drawn from the training data, while during autoregressive inference it can be conditioned on prefixes produced by the model itself.
The important practical point is simpler than the terminology: one generation error changes the context used for later predictions.
A low next-token loss does not guarantee good long sequences
Teacher-forced loss evaluates local predictions under known prefixes. Generation evaluates a process that repeatedly consumes its own decisions.
Those are related, but they are not identical measurements.
Consider a simplified model that usually predicts the correct next token when given a clean reference prefix. If it makes one unusual choice during generation, later predictions are evaluated under a different prefix. Even if each local prediction is usually strong, a long sequence gives more opportunities for the generated path to diverge.
This does not mean teacher-forced next-token loss is useless. It is the natural training objective for many autoregressive models and provides a stable way to measure how well the model predicts held-out tokens under observed prefixes. It does mean that developers should not treat that loss as a complete measurement of end-to-end generation quality.
For a generation system, evaluation should also exercise the model in the way users will run it: start from realistic prompts or initial states, generate autoregressively, and assess the resulting sequence with metrics or human judgments appropriate to the application.
Teacher forcing does not mean the answer is leaked
A common source of confusion is that the full target sequence may exist in the training batch. How can predicting here be meaningful if here is already present somewhere in the tensor?
The answer is the model’s information boundary.
For causal next-token prediction, the position predicting here may use only the earlier prefix:
<BOS> cats sleepIt must not use here or later target tokens as input evidence for that prediction. Causal masking or recurrent ordering enforces this boundary, depending on the architecture.
Teacher forcing gives the model the correct past, not the answer it is currently supposed to predict.
If an implementation accidentally allows a prediction position to access its target token or future tokens, that is target leakage, not a normal consequence of teacher forcing. Training metrics can then look excellent while inference fails because the leaked information is unavailable at generation time.
Error accumulation depends on the task
The training-inference mismatch matters most when later predictions depend strongly on earlier generated decisions.
In open-ended text generation, an alternative early token may still lead to a coherent continuation. There may be many valid sequences rather than one uniquely correct reference. A generated prefix that differs from the dataset is therefore not automatically an error.
In a structured sequence task, an early mistake can be more damaging. Suppose a model generates steps in a machine-control plan:
unlock -> open -> inspect -> closeIf it incorrectly emits close at the second step, later predictions may be conditioned on a state that is invalid for the intended procedure. The cost of compounding errors is much higher when sequence validity is strict.
The same reasoning applies to generated code, structured records, trajectories, and other autoregressive outputs. The practical question is not merely whether teacher forcing was used. Ask how sensitive the application is to prefixes that differ from the training distribution.
Do not confuse teacher forcing with decoding strategy
Teacher forcing describes how previous sequence elements are supplied during training. Decoding describes how predictions are selected during generation.
These are separate choices.
A model trained with teacher forcing might generate with greedy decoding, beam search, temperature sampling, top-p sampling, or another method supported by the system. Changing the decoding strategy can alter the probability of entering unusual prefixes, but it does not remove the underlying difference between ground-truth training prefixes and model-generated inference prefixes.
Similarly, lowering sampling temperature can make generation more concentrated around high-probability choices, but it does not make the model immune to compounding mistakes or distribution shift.
Keep the two layers conceptually separate:
training question:
What previous tokens does the model receive while learning?
inference question:
How do we choose a token from the model's next-token distribution?This separation makes debugging much clearer.
Scheduled sampling changes the training distribution
One proposed response to exposure bias is scheduled sampling. Instead of always supplying the ground-truth previous element during training, the procedure sometimes supplies a model-generated element.
Conceptually:
previous input during training
-> ground-truth token, sometimes
-> model-generated token, sometimesThe motivation is straightforward: expose the model to some of the imperfect prefixes it may encounter during inference.
But this is not a free correction that should automatically replace teacher forcing. Once model-generated tokens enter the training context, the learning problem changes. Training can become harder because later targets may be paired with prefixes that no longer correspond cleanly to the original reference sequence. The appropriate strategy depends on the model, objective, and task.
For many modern autoregressive language-model training pipelines, standard next-token prediction on observed sequences remains the basic training setup. If end-to-end generation quality is poor, first determine whether the problem is actually exposure to generated prefixes rather than assuming scheduled sampling is the remedy.
Diagnose the right failure before changing training
When an autoregressive system performs poorly, several different problems can look like “generation drift.”
If the model makes poor next-token predictions even under held-out ground-truth prefixes, the basic predictive model or data needs improvement. Feeding its own predictions during training does not fix missing knowledge or inadequate capacity.
If teacher-forced validation looks strong but free-running generation degrades after small mistakes, investigate prefix sensitivity. Evaluate generated trajectories, identify where sequences first become invalid or low quality, and inspect how later predictions behave after realistic deviations.
If generation is good until context becomes long, the issue may instead involve context handling, truncation, position behavior, or accumulated task complexity.
If outputs fail only under a particular sampling configuration, decoding may be the more direct place to intervene.
A useful diagnostic sequence is:
1. evaluate next-token predictions on held-out reference prefixes
2. evaluate complete autoregressive generations
3. locate the earliest meaningful divergence or failure
4. test the model on prefixes near that failure
5. change training only if the evidence points to a training-distribution problemThis avoids treating every sequence error as the same phenomenon.
Measure the system the way it will run
A robust evaluation plan normally includes both teacher-forced and free-running views.
Teacher-forced validation can answer questions such as:
- Is next-token predictive performance improving on held-out data?
- Which token types or sequence regions have high loss?
- Did a training change improve the model under comparable reference contexts?
Autoregressive evaluation answers different questions:
- Does the model produce useful complete sequences from realistic starting inputs?
- How often do generated sequences violate task constraints?
- Does quality deteriorate as generated history grows?
- How sensitive are results to the decoding configuration used in production?
The exact metrics depend on the application. A language assistant may require task-specific judgments or reference-based checks; a structured generator may support exact validity tests; a trajectory model may have state-dependent success criteria.
Do not collapse these measurements into one number unless that number genuinely represents the deployment objective.
When teacher forcing is a good fit
Teacher forcing is a natural fit when you have complete training sequences and want to train an autoregressive model to predict each next element from the correct observed history. It provides direct supervised targets at every position and, for suitable architectures, supports efficient computation across many positions.
Be more cautious when the deployed system must recover from its own mistakes in a tightly constrained sequential environment. In that case, reference-prefix performance alone may leave an important part of system behavior untested. You may need training data that covers recovery states, an objective designed for the sequential task, constrained generation, search, reinforcement learning, or another task-specific method. Which intervention is appropriate depends on why the generated trajectories fail.
A simpler non-autoregressive formulation can also be preferable when the problem does not inherently require sequential generation. If the desired output can be predicted independently or represented as a fixed structured decision, introducing a long chain of dependent generation steps can create failure opportunities without adding value.
Conclusion
Teacher forcing trains an autoregressive model by asking for the next element while supplying the correct earlier elements from the training sequence. That gives clean learning contexts and efficient supervision, but inference is different: generated outputs become future inputs.
The reusable mental model is to distinguish prediction under a reference prefix from generation under a model-produced prefix. Measure both when sequence quality matters. When they disagree, locate where generated context begins to cause trouble before changing the training method. That turns “exposure bias” from an abstract label into a concrete debugging question: what prefixes does the model see, and can it make useful predictions from the prefixes it will actually encounter?