Sequence Models

Understand Teacher Forcing in Autoregressive Models

An autoregressive model generates a sequence one element at a time. A language model predicts the next token from the tokens before it; a sequence model might similarly predict the next symbol, event, or value from an existing prefix. That creates a practical training question: when teaching the model to predict step 5, should the input contain the correct steps 1–4 from the dataset, or the model’s own earlier predictions?