Improve LLM Fine-Tuning with Rejection Sampling

Suppose you can tell a good model response from a bad one, but writing thousands of ideal responses by hand is expensive. A capable language model may already produce acceptable answers some of the time. The problem is that those answers are mixed with weaker ones.

Rejection sampling fine-tuning turns that observation into a data-generation loop. For each prompt, generate several candidate responses, evaluate them, keep responses that satisfy a selection rule, and use the accepted prompt-response pairs for supervised fine-tuning. The method can concentrate training on behavior you want without requiring a human to author every target from scratch.

The filtering step is where most of the design work lives. This article builds the method from a small example, explains what the resulting training data actually teaches, and shows how candidate generation, scoring, and selection can quietly bias the final model.

Start with generation, not training

Imagine a model that answers short programming questions. For the prompt:

Explain why a binary search requires sorted input.

you sample four responses. A simplified evaluator gives them these scores on a scale from 0 to 1:

candidate A: 0.92
candidate B: 0.81
candidate C: 0.43
candidate D: 0.18

If your rule is “keep the highest-scoring response,” candidate A becomes the target paired with that prompt. Repeat the process across many prompts and you obtain a new supervised dataset:

prompt 1 -> selected response 1
prompt 2 -> selected response 2
prompt 3 -> selected response 3
...

You can then fine-tune a model with the ordinary next-token objective used for supervised language-model training. The training algorithm itself does not need to know that the targets were selected from generated candidates.

This separation is useful. Rejection sampling is primarily a data selection procedure. It changes which outputs become demonstrations; supervised fine-tuning then increases the likelihood of those selected sequences under the training prompts.

The mental model: search first, imitate second

It helps to think of the process as two distinct stages.

During search, the generator explores several possible responses. The evaluator provides a preference signal by deciding which candidates are worth keeping. During imitation, the fine-tuned model learns from the survivors as if they were ordinary demonstrations.

That distinction explains both the appeal and the main limitation of the method. The selection process can only choose among responses that generation produced. If every candidate misunderstands the prompt, ranking them does not create a correct answer. The best candidate in a weak set can still be bad.

Conversely, if a model occasionally produces excellent responses but not reliably, sampling multiple candidates can expose those good responses often enough to build useful training data. Fine-tuning can then make behavior represented in the selected set more likely.

Rejection sampling therefore depends on two capabilities that should be evaluated separately:

1. Candidate coverage: does generation produce responses with the desired behavior often enough?
2. Selection quality: can the evaluator reliably distinguish those responses from weaker ones?

A failure in either stage contaminates the training set.

Selection rules define the dataset

“Rejection sampling” does not imply one universal acceptance rule. Several policies are possible, and they produce different data.

Keep the best candidate per prompt

A common design generates n candidates and retains the one with the highest score:

selected = argmax(score(candidate_i))

This guarantees at most one target per prompt and is easy to reason about. Increasing n gives the selection step more opportunities to find a strong response, but generation cost grows with the number and length of candidates.

There is another subtle effect: best-of-n selection becomes more aggressive as n grows. The chosen examples increasingly reflect whatever the scorer rewards most strongly. If the scorer has a systematic weakness, a larger candidate pool gives generation more chances to exploit it.

Keep every candidate above a threshold

Another policy accepts any response whose score exceeds a threshold:

accept candidate if score(candidate) >= 0.85

This can preserve multiple valid ways to answer the same prompt. It also means some prompts may contribute several examples while others contribute none. Without care, prompts for which the generator performs well can become overrepresented in the fine-tuning set.

A threshold only has meaning relative to the scorer. A value such as 0.85 is not intrinsically “high quality.” You need to inspect how scores correspond to actual acceptance criteria on representative data.

Apply hard checks before ranking

Some requirements are better represented as gates than as small score differences. If an answer must be valid JSON, compile successfully, avoid a prohibited field, or contain a verifiable final value, check that condition directly when possible.

A practical pipeline might be:

candidates
    -> deterministic validity checks
    -> task-quality scoring
    -> select among remaining candidates

This prevents a high aggregate score from compensating for a requirement that should have been mandatory.

Generation settings change what selection can find

The candidate generator is part of the data pipeline. Sampling settings that are useful for a user-facing assistant are not necessarily the right settings for producing training candidates.

If generation is nearly deterministic, repeated samples may be almost identical. Generating eight copies of the same mistake gives the selector little value. More diverse sampling can expose alternative reasoning paths, wording, or solutions, but excessive randomness may spend most of the budget on low-quality candidates.

The useful operating point depends on the model and task. Rather than treating temperature or candidate count as fixed recipes, measure at least two quantities: how often the pool contains an acceptable answer and how much additional generation each improvement costs.

Candidate length matters too. If the evaluator tends to reward detailed responses, unconstrained generation may gradually select longer targets even when shorter answers solve the task. That changes the style the fine-tuned model sees. Length limits or explicit concision criteria can be appropriate when verbosity is not itself the objective.

Keep generation metadata during dataset construction. Recording the model version, decoding settings, candidate count, evaluator version, and raw score makes later failures much easier to trace than a dataset containing only the final accepted text.

The evaluator becomes part of the training objective

The fine-tuning loss does not directly optimize the evaluator’s score. Even so, the evaluator shapes training by controlling which examples survive.

Suppose a scorer rewards answers that contain a correct final result but does not inspect the reasoning that leads there. A response with invalid reasoning and a lucky final answer may pass. Once selected, supervised fine-tuning treats the entire response as a target, including the flawed reasoning.

The same issue appears with style. A scorer that strongly rewards a particular format can cause the selected dataset to collapse toward that format, even when several formats would serve users equally well.

For that reason, evaluator validation should happen before large-scale candidate generation. Take a representative sample of candidate pairs or pools and check whether the evaluator’s ordering agrees with the actual criteria you care about. Pay special attention to near-ties, unusual lengths, malformed outputs, and examples that look as though they could satisfy the metric without satisfying the task.

An automated evaluator can be a model, deterministic program, task-specific verifier, or combination of these. Each provides different evidence. A compiler can verify syntax and some semantics but cannot judge whether an explanation is pedagogically clear. A language-model judge can assess broader qualities, but its scores should not be treated as ground truth merely because they are numeric.

Avoid training only on easy prompts

Filtering can change the prompt distribution even when you never intended to change it.

Assume the original prompt set contains equal numbers of easy and difficult tasks. If an 0.85 threshold accepts 90% of easy-task candidates but only 20% of difficult-task candidates, the final dataset will be dominated by easy tasks. Fine-tuning on it may improve the region where the model was already strongest while providing little signal for the failures you wanted to fix.

Track acceptance rates by meaningful slices such as task type, difficulty, language, input length, or source. Large differences do not automatically mean the pipeline is wrong, but they tell you what distribution the model will actually imitate.

Possible responses include generating more candidates for difficult prompts, using different selection rules by task where justified, retaining prompt-level sampling weights, or collecting human-written targets for cases where generation rarely succeeds.

Do not lower a quality threshold blindly just to balance counts. If no generated candidate is good enough, admitting a bad one makes the dataset larger while weakening its supervision.

Prevent the generator and evaluator from hiding each other’s failures

Using the same model family for generation and evaluation can be convenient, but shared weaknesses can matter. A generator may repeatedly produce a misconception that a closely related evaluator also fails to notice. The resulting dataset can look internally consistent while preserving the error.

Independence is not binary; the practical question is whether your checks catch the failure modes that matter. For tasks with executable answers, external verification can provide a different signal. For factual or domain-sensitive work, curated test cases and human review may be necessary. For subjective writing tasks, multiple evaluation criteria can reveal trade-offs that a single scalar score hides.

Also keep a held-out evaluation set outside the rejection-sampling loop. If the same prompts or reference cases repeatedly drive selection decisions and final evaluation, improvements can reflect adaptation to the data-construction procedure rather than better behavior on new inputs.

Understand the cost trade-off

Rejection sampling moves some cost from human target creation to model inference and evaluation.

If there are P prompts and you generate n candidates for each, candidate generation requires P * n completions before filtering. The exact compute cost depends on input lengths, generated lengths, model architecture, batching, hardware, and serving implementation. Evaluator calls can add another substantial term when scoring requires model inference.

More candidates are useful only while the additional search improves the selected dataset enough to justify that cost. If acceptable responses already appear in nearly every small candidate pool, doubling n may add little. If acceptable responses are extremely rare, even a large n may be an inefficient substitute for improving the base model or collecting better targets.

There is also a training-data cost that is easy to miss. Keeping several near-duplicate winners does not provide the same value as adding genuinely different, high-quality prompts. Before spending heavily on larger candidate pools, inspect whether the bottleneck is response quality or prompt coverage.

Common mistakes

A few mistakes recur because the pipeline looks simpler than it is.

Treating the top score as proof of correctness. Ranking identifies the best candidate according to the evaluator. It does not establish an absolute quality level. Use minimum requirements when “least bad” is not good enough.

Discarding rejected candidates too early. You may not train on them, but retaining a sampled subset is useful for diagnosing why candidates fail and whether the evaluator separates good and bad responses sensibly.

Ignoring duplicates. Repeated prompts or nearly identical accepted responses can give a narrow behavior disproportionate weight. Deduplication should preserve meaningful distinctions rather than relying only on exact string equality.

Changing the scorer without versioning the dataset. A new evaluator changes the acceptance boundary. Record enough provenance to know which rule produced each dataset version.

Evaluating only the fine-tuning loss. Lower loss on selected responses shows that the model is fitting those targets. It does not show that the deployed model became more correct, robust, or useful. Evaluate the behavior you intended to improve on held-out prompts.

When rejection sampling fine-tuning fits

The method is a good candidate when your current model can already generate desirable outputs with reasonable frequency, you have a credible way to recognize them, and supervised fine-tuning is an acceptable way to make those patterns more common.

It is less attractive when correct outputs are almost absent from the candidate distribution, evaluation is too unreliable to separate good responses from convincing failures, or the desired behavior cannot be captured well by selecting complete target sequences. In those cases, better source data, stronger verification, a more capable generator, or a different training objective may address the bottleneck more directly.

Rejection sampling can also be used iteratively: generate with a model, select data, fine-tune, then generate a new dataset with the improved model. Iteration can increase candidate quality, but it can also amplify evaluator bias and reduce diversity. Re-evaluate the data distribution and held-out behavior at every round rather than assuming the loop improves itself automatically.

Build the loop around measurable failures

A useful rejection sampling pipeline starts with a concrete failure you can recognize. Define what an acceptable response means, test whether the generator can produce such responses, and validate that the selector actually identifies them. Only then is it worth scaling candidate generation and fine-tuning.

The core idea is deliberately modest: search over outputs, keep evidence of better behavior, then teach the model from what survived. Its quality comes from the details around that loop. Candidate diversity determines what can be found, the evaluator determines what counts as good, and filtering determines which parts of the prompt distribution remain visible. Measure all three, and rejection sampling becomes a controlled data-improvement method rather than a machine for turning scores into unquestioned labels.