A language model can learn to imitate examples with supervised fine-tuning, but imitation alone does not directly express a common requirement: for the same prompt, one acceptable response may be preferable to another.

Preference data represents that requirement as comparisons. A training record contains a prompt, a chosen response, and a rejected response. Direct preference optimization (DPO) uses those pairs to adjust a language model so that the chosen response becomes more favored relative to the rejected one, while comparing the update with a fixed reference model.

DPO is useful because it turns preference learning into a classification-like training objective rather than requiring a separately trained reward model and an online reinforcement-learning loop. That does not make preference alignment simple: data quality, reference-model choice, response length, distribution shift, and evaluation still matter.

This article builds the DPO mental model from one preference pair, explains the objective without hiding the important terms, and shows what developers should measure before treating a lower training loss as an improvement.

Start with one comparison

Suppose a coding assistant receives this prompt:

Explain why a database transaction might deadlock.

A preference dataset might contain:

chosen:  "Two transactions can deadlock when each holds a lock the other needs..."
rejected: "A deadlock happens whenever two transactions run at the same time."

The label does not say that the chosen response is perfect. It says only that, for this prompt and according to the preference source, the chosen response is preferred to the rejected response.

That distinction is fundamental. DPO learns from relative comparisons. If both responses are poor, preferring the less poor one does not turn it into a high-quality target. If both are excellent, the comparison may encode a subtle style preference rather than factual quality.

A useful way to picture one training example is:

prompt x
   |-- chosen response y_w
   `-- rejected response y_l

policy:    should favor y_w over y_l
reference: supplies the baseline for that preference change

Here w means winner and l means loser within the pair.

Compare response probabilities, not individual tokens

An autoregressive language model assigns a conditional probability to a complete response by multiplying its next-token probabilities. In practice, log probabilities are easier to work with because products become sums.

For a response containing tokens y_1 ... y_T:

log pi(y | x) = sum_t log pi(y_t | x, y_<t)

For the chosen and rejected responses, define the policy’s log-probability margin:

policy_margin = log pi(y_w | x) - log pi(y_l | x)

A positive margin means the policy assigns the chosen response higher sequence probability than the rejected response. A larger margin means a stronger relative preference under this measure.

DPO does not optimize that policy margin in isolation. It compares it with the same margin under a fixed reference policy, usually a model representing the behavior that training should not drift away from without evidence from the preference data:

reference_margin = log pi_ref(y_w | x) - log pi_ref(y_l | x)

The important quantity is therefore:

advantage = policy_margin - reference_margin

If advantage is positive, the trainable policy prefers the chosen response over the rejected response more strongly than the reference does.

Turn the comparison into the DPO loss

For one pair, the standard DPO objective can be written as:

loss = -log sigmoid(beta * advantage)

where beta is a positive scaling parameter and:

sigmoid(z) = 1 / (1 + exp(-z))

Consider a simplified numerical example:

policy_margin    =  1.2
reference_margin =  0.4
beta             =  0.5

Then:

advantage = 1.2 - 0.4 = 0.8
z         = 0.5 * 0.8 = 0.4
sigmoid(z) ~= 0.599
loss       ~= 0.513

Now imagine the policy margin were -0.2 while the reference margin remained 0.4:

advantage = -0.6
z         = -0.3
sigmoid(z) ~= 0.426
loss       ~= 0.854

The second pair produces a larger loss because the policy has moved in the wrong direction relative to the reference baseline.

The value of beta changes the scale of the logit passed to the sigmoid and is part of the objective’s trade-off with the reference policy. Its practical effect should be validated for the implementation and dataset being used rather than interpreted as a universal quality knob.

Why the reference model is part of the signal

It is tempting to describe DPO as simply “increase the chosen response and decrease the rejected response.” That description misses an important detail.

The loss depends on differences of log-probability ratios relative to the reference. The policy is rewarded for changing the chosen-versus-rejected relationship in the preferred direction compared with that baseline.

For example, suppose the reference already strongly favors the chosen answer:

reference_margin = 3.0
policy_margin    = 3.1

The policy margin is large, but the DPO advantage is only 0.1. By contrast:

reference_margin = -1.0
policy_margin    = 0.5

produces an advantage of 1.5. The second policy has made a much larger preference change relative to its reference even though its absolute policy margin is smaller.

This is why the reference model is not merely an implementation artifact. Changing it changes the training objective.

Compute sequence log probabilities carefully

A practical implementation needs the conditional log probability of each response under both the trainable policy and the reference policy. Conceptually:

for each (prompt, chosen, rejected):
    p_c = policy_logprob(prompt, chosen)
    p_r = policy_logprob(prompt, rejected)

    q_c = reference_logprob(prompt, chosen)
    q_r = reference_logprob(prompt, rejected)

    advantage = (p_c - p_r) - (q_c - q_r)
    loss = -log_sigmoid(beta * advantage)

This pseudocode is deliberately framework-independent. Production libraries differ in batching, masking, tokenization, numerical stabilization, and whether reference log probabilities are precomputed.

Several details must remain consistent:

  • Score response tokens conditionally on the prompt; do not accidentally include prompt tokens in the response loss unless the chosen implementation explicitly defines that behavior.
  • Use compatible tokenization and model inputs for policy and reference scores.
  • Mask padding so it does not contribute token log probabilities.
  • Keep the reference policy fixed for standard DPO. Updating it together with the policy changes the objective.
  • Use numerically stable log-sigmoid operations rather than manually computing log(sigmoid(z)) when the framework provides a stable primitive.

These are implementation details, but mistakes in them alter the quantity the mathematical objective assumes you are optimizing.

Preference data determines what the model learns

DPO cannot recover a preference signal that is absent or inconsistent in the dataset. A pair can encode factuality, helpfulness, formatting, tone, safety behavior, verbosity, or an accidental artifact.

Suppose chosen answers in a dataset are usually longer than rejected answers. A model may learn that length is predictive of preference even when length is not the intended criterion. Similar shortcuts can arise from phrases, formatting patterns, source-specific wording, or systematic differences between response generators.

Before training, inspect pairs for questions such as:

Can a reviewer explain why the chosen response wins?
Would the preference remain the same if superficial formatting changed?
Are both responses relevant to the same prompt?
Are there many near-duplicate pairs?
Does one data source dominate the labels?

When preferences are subjective, disagreement is expected. The goal is not necessarily to eliminate disagreement, but to understand whether the dataset represents the behavior the product intends to optimize.

Relative preference creates important failure modes

Because the objective is pairwise, several outcomes can look successful in training while being undesirable in generation.

A chosen response can still be bad

If the pair is “bad versus worse,” DPO receives no third option saying that neither response is acceptable. Filtering and constructing preference pairs therefore matters as much as optimizing them.

Pair accuracy can hide generation quality

A model can improve the fraction of held-out pairs for which it scores the chosen response above the rejected response without becoming better on free-form prompts. Pairwise scoring evaluates the comparisons you supplied; generation introduces a much larger space of possible outputs.

The training distribution can be too narrow

If preference prompts cover only coding questions, an improvement on those pairs says little about unrelated tasks. Updates can also change behavior outside the targeted domain, so regression evaluation should include capabilities that must be preserved.

Sequence probability depends on length

A sequence log probability is a sum of token log probabilities. Longer responses contain more terms in that sum. DPO implementations and variants may handle length-related effects differently, and preference datasets can contain length biases. Treat response length as something to measure, not as an invisible nuisance variable.

Evaluate the behavior, not just the loss

A useful DPO evaluation has multiple layers.

First, keep a held-out preference set and measure whether the trained policy improves the intended pairwise ordering. This catches basic failures to learn the supplied signal.

Second, evaluate generated responses on prompts that represent actual use. Depending on the application, evaluation may include blinded human comparisons, carefully designed model-assisted judging with validation against human decisions, task-specific correctness checks, or deterministic tests for requirements such as valid structured output.

Third, run regression evaluations. If the alignment dataset targets concise support answers, for example, verify that the model has not degraded on factual questions, instruction following, code tasks, or other capabilities the application relies on.

Also monitor behavioral indicators related to the preference data itself, such as response length and refusal rate. A metric moving strongly even though it was not an intended training target can reveal a shortcut in the data.

When DPO is a good fit

DPO is a practical option when you already have a suitable language model, can collect meaningful offline preference pairs, and want to optimize relative response preferences without building a reward-model-plus-online-RL training pipeline.

A simpler method may be preferable in other cases. If you have only high-quality target responses and no meaningful rejected alternatives, supervised fine-tuning matches the available signal more directly. If the desired behavior can be specified reliably at inference time, prompting may avoid another training stage. If the task has an objective executable reward and requires exploration beyond a fixed offline comparison set, a different optimization approach may better match the problem.

The choice should follow the signal you actually possess, not the popularity of an alignment method.

Conclusion

Direct preference optimization converts chosen-versus-rejected response pairs into a training signal by comparing the policy’s preference margin with the same margin under a fixed reference model. The core mental model is simple: learn to favor the chosen response more than the reference does, not merely to assign the chosen response a high probability in isolation.

That relative objective makes DPO convenient, but it also defines its limits. Pair quality, reference choice, sequence scoring, length effects, distribution coverage, and generation-time evaluation all influence whether a lower DPO loss becomes a better model. Treat preference optimization as one stage in a measured evaluation loop, not as a substitute for defining and testing the behavior you want.

Artificial Intelligence Language Models Fine-Tuning Model Alignment