When an AI application produces weak results, teams often jump directly to fine-tuning. That can be the right choice, but many problems are cheaper and easier to solve with better prompting or retrieval-augmented generation (RAG).

The three approaches change different parts of the system. Prompting changes the instructions and context given at inference time. RAG supplies relevant external information at inference time. Fine-tuning changes the model’s learned parameters through additional training.

Choosing between them starts with identifying what is actually missing: clearer behavior, better knowledge, or a learned pattern the base model does not reliably follow.

Start with prompting

Prompting is usually the lowest-cost intervention because it does not require a training pipeline or a retrieval system. A good prompt can define the task, constraints, output format, examples, and criteria for success.

For example, an extraction task may improve substantially when the prompt specifies an exact schema and explains what to do when a field is unknown. A support assistant may become more consistent when its system instructions define tone, escalation rules, and prohibited actions.

Prompting is a strong fit when:

  • the model already has enough capability and knowledge for the task;
  • failures come from ambiguous instructions;
  • requirements change frequently;
  • examples fit comfortably within the available context window;
  • you need a fast iteration cycle.

Prompting has limits. Repeating many examples consumes context, and instructions cannot reliably add private or newly updated knowledge that the model has never received.

Use RAG when the missing piece is knowledge

RAG retrieves relevant information from an external source and places it in the model’s context before generation. The model can then answer using information that was not stored in its parameters during training.

A typical flow looks like this:

user question
    |
    v
retrieve relevant documents
    |
    v
build prompt with retrieved context
    |
    v
language model
    |
    v
answer

RAG is especially useful for product documentation, internal policies, knowledge bases, catalogues, and other information that changes independently of the model.

Choose RAG when:

  • answers depend on private or frequently changing information;
  • users need responses grounded in specific source material;
  • the knowledge set is too large to place entirely in every prompt;
  • content must be updated without retraining the model.

RAG does not automatically make answers correct. Retrieval can return irrelevant, incomplete, or outdated passages. Chunking, indexing, retrieval quality, context construction, and answer evaluation all remain important.

Fine-tune when you need learned behavior

Fine-tuning continues training a model on examples chosen for a particular objective. Depending on the method, training may update all model parameters or only a smaller set of additional or selected parameters.

Fine-tuning is most useful when the desired improvement is a repeatable behavior that is difficult or expensive to express through prompts alone. Examples include a specialised response style, domain-specific classification patterns, consistent transformations, or task formats demonstrated by a substantial set of high-quality examples.

Consider fine-tuning when:

  • strong prompts still produce inconsistent task behavior;
  • you have enough representative training examples;
  • the target behavior is relatively stable;
  • reducing long few-shot prompts would materially improve latency or token cost;
  • evaluation shows that training improves the target task rather than merely changing its style.

Fine-tuning should not be treated as a reliable way to keep factual knowledge current. Facts encoded during training can become stale, and the model may not reproduce every trained fact accurately. If information changes regularly, retrieval is usually a better mechanism for supplying it.

Separate knowledge problems from behavior problems

A useful diagnostic question is: Would the model succeed if the correct information were placed directly in its prompt?

If the answer is yes, the core problem is probably access to knowledge. RAG may be appropriate.

If the model receives the necessary information but still performs the transformation, classification, or response pattern poorly, the problem may be behavioral. Better prompting should come first, followed by fine-tuning if measured results justify it.

If neither knowledge nor learned behavior is missing, adding RAG or fine-tuning can create complexity without solving the actual issue. The problem may instead be poor task definition, weak evaluation, unsuitable model capability, or an application workflow that expects too much from a single generation step.

Combine the approaches when their roles are clear

Prompting, RAG, and fine-tuning are not mutually exclusive.

A production system might use:

fine-tuned model
    +
clear system instructions
    +
retrieved current documentation
    =
application response

Each component should have a specific responsibility. The prompt defines the current task and constraints. Retrieval supplies current evidence. Fine-tuning provides stable learned behavior.

This separation also makes debugging easier. If an answer contains an outdated policy, inspect retrieval and source freshness. If the answer ignores a formatting rule, inspect the prompt and learned behavior. Without clear boundaries, teams can end up retraining models to compensate for retrieval failures or expanding prompts to compensate for poor training data.

Compare cost beyond the model call

The cheapest-looking option at inference time is not always cheapest overall.

Prompting requires little infrastructure, but very long prompts increase token usage and latency. RAG adds document ingestion, indexing, retrieval, access control, and monitoring. Fine-tuning adds dataset preparation, training, model versioning, evaluation, and potentially separate serving requirements.

Think about total operational cost:

Approach Main ongoing work Easy to update? Best suited to
Prompting Prompt testing and versioning Yes Instructions and task framing
RAG Indexing and retrieval quality Yes Dynamic or private knowledge
Fine-tuning Dataset and model lifecycle Usually slower Stable learned behavior

The exact economics depend on the model provider, traffic, context size, training method, and serving architecture. Measure with your own workload rather than assuming one approach is always cheaper.

Evaluate before adding complexity

Create a representative evaluation set before changing the architecture. Record the current system’s results, identify failure categories, and measure each proposed intervention against the same examples.

A practical sequence is:

  1. establish a baseline with a clear prompt;
  2. improve the prompt and measure again;
  3. add retrieval if failures are caused by missing knowledge;
  4. fine-tune only when a persistent behavioral gap remains and suitable training data exists;
  5. re-evaluate the complete system, including failure cases.

This order is not a rigid rule. Some applications clearly require private retrieval or specialised training from the beginning. The principle is to introduce complexity because evidence supports it, not because a technique is fashionable.

Watch for data leakage and evaluation contamination

Fine-tuning and RAG both depend heavily on data quality. Training examples should not leak secrets, private user information, or evaluation answers into places where they do not belong. Retrieved content should respect the same authorisation boundaries as the application itself.

Keep evaluation examples separate from fine-tuning data when you want an unbiased estimate of generalisation. If the model trains directly on the test cases, improved scores may reflect memorisation rather than better performance on unseen inputs.

For RAG, also test questions whose answers are absent from the knowledge base. A useful system should handle missing evidence safely rather than inventing a confident answer.

Use the smallest intervention that solves the measured problem

Prompting, RAG, and fine-tuning operate at different layers of an AI system. Prompting tells the model what to do now. RAG gives it information to use now. Fine-tuning changes patterns the model has learned.

Start by classifying the failure. If instructions are unclear, improve the prompt. If knowledge is missing or changes frequently, retrieve it. If a stable behavior remains unreliable despite good instructions and sufficient context, evaluate fine-tuning.

That approach keeps the architecture understandable and makes each additional component earn its operational cost.

Artificial Intelligence LLM RAG Fine-Tuning