Prefix Tuning

Adapt Language Models with Prefix Tuning

Adapt Language Models with Prefix Tuning Full fine-tuning changes a model’s weights for each task. That can be effective, but storing and serving a separate full checkpoint for every task becomes expensive as model size and task count grow. Prefix tuning offers a different arrangement: keep the pretrained model frozen and train a small set of task-specific states that participate in attention. This article builds a practical mental model for prefix tuning, shows how it differs from text prompts and low-rank weight adapters, and explains the trade-offs that matter when training or serving several task variants.