A language model can produce several plausible answers to the same prompt. That variability is often treated as noise, but it can also be used deliberately: generate multiple candidates, evaluate them, and return the strongest one.
This pattern is called best-of-N sampling. Instead of trusting one generation, the system samples N responses and uses a scoring rule to select one. The extra samples spend more compute at inference time in exchange for more opportunities to find a good response.
The key limitation is easy to miss: best-of-N is only as useful as the selection process. Generating more candidates does not guarantee a better final answer, and a flawed scorer can become more damaging as it gets more candidates to optimize over. This article builds the method from a small example, explains its cost and quality trade-offs, and shows when simpler decoding is the better choice.
Start with generate, score, select
Suppose an assistant must answer a debugging question:
Why can retrying a non-idempotent payment request create duplicate charges?Instead of generating one answer, the system samples four:
A -> score 0.62
B -> score 0.91
C -> score 0.74
D -> score 0.55If the scores are trustworthy for this task, the system returns candidate B.
The algorithm is conceptually simple:
candidates = generate(prompt, n=4)
scored = [(candidate, score(prompt, candidate)) for candidate in candidates]
return candidate_with_highest_score(scored)This is teaching pseudocode, not a claim about a particular model API. Some APIs can request several completions in one call; others require separate requests or application-level batching.
The important separation is between two jobs:
- the generator creates possible responses;
- the scorer decides which candidate is preferred.
Best-of-N improves the search over responses only when those two parts work together well.
Why multiple samples can help
Sampling from a language model does not repeatedly produce the same continuation unless decoding is made deterministic. Different samples can take different reasoning paths, choose different explanations, or make different mistakes.
Imagine that, for a particular class of prompts, a single sample has some chance of producing a response that satisfies the application’s quality criteria. Drawing several sufficiently diverse samples creates more opportunities for at least one acceptable candidate to appear.
That does not mean the probability of success follows a simple formula in a real system. Samples from the same model are not necessarily independent in the practical sense that matters: they share the same prompt, model, training biases, and decoding setup. A prompt that systematically misleads the model may produce the same conceptual error across many samples.
The useful mental model is therefore not “four samples are four times as good.” It is:
more samples -> broader candidate set -> more selection opportunitiesWhether those opportunities improve the returned response depends on candidate diversity and scorer quality.
The scorer defines what “best” means
Best-of-N has no universal definition of quality. The selection rule supplies one.
For a coding assistant, a candidate might be scored using executable tests. For a structured extraction task, the system might check schema validity and compare extracted fields against deterministic constraints. For open-ended dialogue, selection may require a learned reward model, another model acting as a judge, human preference, or a combination of signals.
These scoring methods have very different reliability properties.
A deterministic test can be strong evidence for the behavior it actually tests, but passing tests do not prove that code is secure, maintainable, or correct for untested inputs. A learned reward model can evaluate qualities that are difficult to encode as rules, but its score is a model prediction rather than an objective guarantee. An LLM judge can compare natural-language answers flexibly, but it can have its own biases and failure modes.
Treat the scorer as part of the product behavior, not as an invisible implementation detail. If the scorer rewards the wrong thing, best-of-N searches harder for the wrong thing.
Separate candidate generation from evaluation
A useful implementation makes the two stages observable.
For each request, record enough non-sensitive metadata to answer questions such as:
How many candidates were generated?
How different were their scores?
Why did the winner pass required checks?
How often would candidate 1 have been selected anyway?This separation helps diagnose whether additional inference compute is buying anything.
Suppose the winning score for five requests looks like this:
request N=1 score best of 4
1 0.71 0.83
2 0.88 0.89
3 0.42 0.45
4 0.76 0.92
5 0.81 0.81The example shows three different situations. Requests 1 and 4 have useful alternatives. Request 2 changes little. Request 3 remains weak even after more sampling, which may indicate that the prompt, model capability, retrieved context, or scorer needs attention instead of a larger N.
Do not interpret these made-up scores as calibrated probabilities. They are only a compact example of how per-request measurements can reveal diminishing value.
Choose sampling settings that create useful alternatives
Best-of-N needs variation. If every candidate is nearly identical, generating more of them mostly repeats the same work.
Temperature, top-p sampling, and other decoding controls can affect diversity, but their exact behavior and available parameters depend on the model and inference implementation. Increasing randomness can broaden the candidate set, while too much randomness can fill it with low-quality responses.
That creates a three-way interaction:
decoding diversity <-> number of samples <-> scorer reliabilityA stronger scorer may be able to benefit from a wider candidate distribution because it can reject poor alternatives. A weak scorer may perform worse when given increasingly unusual candidates that exploit its blind spots.
Tune these settings on representative tasks rather than assuming that a larger N or higher temperature is automatically better.
Account for the real inference cost
Generating N complete candidates usually requires substantially more generation work than producing one. The exact cost does not have to grow perfectly linearly: batching, shared prompt processing, hardware utilization, candidate lengths, and serving architecture all affect latency and throughput.
Still, best-of-N spends extra compute by design. The system may pay for:
- additional generated tokens;
- memory for concurrent candidate states;
- scorer inference or external checks;
- longer end-to-end latency if candidates or scores cannot be computed in parallel;
- reduced serving capacity when the same hardware could have handled other requests.
Measure both user-facing latency and total compute or token cost. Parallel generation can hide some wall-clock latency while still consuming much more capacity.
The scoring stage also matters. If a large judge model evaluates every candidate, selection can cost as much as or more than generation. A cheap deterministic verifier can make best-of-N much more attractive for tasks where such a verifier is meaningful.
Expect diminishing returns
Increasing N cannot create capabilities that are effectively absent from the generator’s candidate distribution. If useful answers are already common, a few samples may capture most of the benefit. If useful answers are extremely rare, adding a small number of samples may barely help.
Selection also has a ceiling. Once the candidate set commonly contains a good answer, further improvement depends increasingly on distinguishing among already competitive responses.
For that reason, evaluate several budgets such as N = 1, 2, 4, 8 on the same held-out workload. Compare final task quality against latency and cost. The useful operating point is where the marginal quality improvement still justifies the marginal inference expense for the application.
Do not select N from scorer values alone. Measure the final metric users care about, such as test pass rate, verified task success, human preference, or another task-specific outcome.
Watch for scorer overoptimization
The most important failure mode is selection against an imperfect proxy.
Suppose a judge tends to reward confident, detailed answers even when some details are unsupported. With one candidate, that bias may cause occasional mistakes. With many candidates, the selection process gets more chances to find a response that scores unusually well under the judge’s preference for confidence.
The highest-scoring candidate can therefore be an extreme scorer favorite without being the most correct response.
This is a general optimization problem: when a proxy metric is imperfect, optimizing it more aggressively can expose its weaknesses. Best-of-N increases optimization pressure simply by comparing more alternatives.
Mitigations depend on the task, but useful practices include validating scorer decisions against an independent evaluation set, keeping hard deterministic checks separate from soft preference scores, inspecting high-scoring failures, and testing performance as N increases. If measured task quality stops improving while scorer values continue rising, treat that divergence as a warning rather than a success.
Do not confuse best-of-N with majority voting
These methods both generate multiple responses, but they use the set differently.
Best-of-N assigns or derives a quality signal for individual candidates and selects the highest-ranked one. Majority voting groups candidates by an answer or decision and selects the most common result.
For a multiple-choice or numeric reasoning task, several independently sampled solutions may converge on the same final answer, making voting meaningful. For an open-ended explanation, exact voting is less natural because many different wordings can all be correct.
The methods can also be combined. A system could group equivalent final answers, use agreement as one signal, and then score representative explanations. But the extra complexity should solve an observed problem rather than being added by default.
Use adaptive budgets when requests have different difficulty
A fixed N is easy to operate, but not every request benefits equally from extra samples.
A system can sometimes stop early when it has strong task-specific evidence that a candidate is sufficient. For example, code generation might stop once a candidate passes a comprehensive deterministic validation suite. Another system might generate an initial small batch and spend more compute only when candidate scores are close or all candidates fail a required check.
Adaptive policies can reduce average cost, but their stopping rule needs evaluation too. A high scorer value is not automatically evidence that the answer is correct, especially when the scorer is poorly calibrated or vulnerable to the same errors as the generator.
Start with a fixed budget because it is easier to benchmark. Add adaptive behavior only after the measurements show where extra samples are being wasted.
Know when a simpler approach is better
Best-of-N is a good fit when candidate quality varies meaningfully, the application has a credible way to rank or verify candidates, and the value of better selection justifies additional inference cost.
It is less attractive when one generation already succeeds reliably, latency or throughput is tightly constrained, candidates are nearly identical, or no trustworthy selection signal exists. It is also the wrong fix when failures come from missing information. If an answer requires a document the model never received, retrieval or tool use addresses the cause more directly than repeatedly sampling from the same incomplete context.
Likewise, if a deterministic transformation can solve the task, use it. Sampling several language-model outputs and judging them is unnecessary complexity when a parser, database query, or ordinary program can produce the required result reliably.
Treat inference compute as a budget, not a guarantee
Best-of-N sampling turns generation into a small search problem: create several possible responses, evaluate them, and keep one. The method can improve final quality when useful alternatives appear and the scorer can recognize them.
Its practical value comes from measuring the whole loop. Track candidate diversity, final task quality, scorer errors, latency, token use, and serving capacity as N changes. Pay particular attention to cases where the scorer becomes more enthusiastic but independent evaluation does not improve.
More inference compute gives the system more choices. It does not make those choices correct by itself. The selection rule is what turns extra samples into useful work—or into extra cost.