https://nalar.dev/tags/model-alignment/ Recent content in Model Alignment on Nalar Hugo en-us Sat, 05 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-direct-preference-optimization-for-language-models/ Sat, 05 Sep 2026 00:00:00 +0700 https://nalar.dev/understand-direct-preference-optimization-for-language-models/ <p>A language model can learn to imitate examples with supervised fine-tuning, but imitation alone does not directly express a common requirement: for the same prompt, one acceptable response may be preferable to another.</p> <p>Preference data represents that requirement as comparisons. A training record contains a prompt, a <strong>chosen</strong> response, and a <strong>rejected</strong> response. <strong>Direct preference optimization (DPO)</strong> uses those pairs to adjust a language model so that the chosen response becomes more favored relative to the rejected one, while comparing the update with a fixed reference model.</p>