RAG vs Fine-Tuning vs Long Context: When Each Actually Wins

The choice that quietly decides whether your AI product works

Every team building with large language models hits the same fork within a week of starting. Should we retrieve documents at query time, fine-tune the model on our data, or just paste everything into the prompt?

Get this wrong and you ship something slow, expensive, or confidently wrong. This guide is the short version: three definitions, named tradeoffs in plain numbers, and a decision tree you can apply on Monday morning without rereading a paper.

The three things, defined once

Retrieval-augmented generation — RAG, coined in a 2020 paper by Lewis et al. at Facebook AI Research (now Meta FAIR) — bolts a search engine onto a language model. When the user asks a question, the system retrieves relevant chunks from a vector database (a store that indexes documents by semantic meaning, not keywords), pastes them into the prompt, and lets the model answer using that context. The model's weights never change. Perplexity, Glean, and most serious customer-support bots work this way.

Fine-tuning modifies the weights of the model itself. You take a base model — Llama 3, GPT-4o-mini, Mistral — and continue training it on your own examples. The model literally learns to behave differently. LoRA (Low-Rank Adaptation, Hu et al. 2021) made this affordable by training a small adapter layer instead of the full network: a fine-tune that used to cost $100,000 of compute now costs closer to $50.

Long context is the lazy option, and increasingly the smart one. You stuff your documents directly into the prompt and let the model read them every time. Gemini 1.5 takes 1 million tokens (around 750,000 words, or two long novels). Claude 3 takes 200,000. GPT-4 Turbo takes 128,000. Two and a half years ago, the GPT-3.5 default was 4,000.

The temptation is to treat these as competitors. They are not. They solve different problems, badly disguised as the same problem.

When RAG wins

RAG is the right answer when your knowledge is large, changes often, or needs to be cited. Three concrete signals.

The corpus does not fit in context. A law firm has forty years of case files. A hospital has every internal protocol since 1998. Even Gemini's million-token window cannot hold a serious enterprise knowledge base, and you would be paying input tokens on every single query whether the user needed those documents or not.

The knowledge changes. A SaaS company's product docs update every sprint. A news desk has new articles every hour. Fine-tuning on this data freezes it; you would be retraining nightly. RAG just re-indexes, which is a one-line operation on most vector databases.

You need attribution. Regulated industries — medicine, law, finance — and any product where the user asks "where did you get that?" need source links. RAG gives you that for free: you retrieved chunk X, you cite chunk X. A fine-tuned model has no idea where its knowledge came from. It cannot tell you, and it will confidently invent a citation if you ask, which is how Steven Schwartz, a New York lawyer, ended up in front of a federal judge in June 2023 having submitted six entirely fictional cases generated by ChatGPT.

The tradeoff: RAG adds a hop. Every query goes through retrieval first, so latency rises 200 to 800 milliseconds, and your system has more moving parts. Bad retrieval means bad answers, and "improving retrieval quality" is its own multi-month engineering project that nobody warns you about at the start.

When fine-tuning wins

Fine-tuning is for behaviour, not knowledge. The mental model: RAG teaches a model what to know, fine-tuning teaches it how to act.

Style and format. If every output must be a JSON object with specific fields, every email must sound like your brand, or every code completion must use your internal API conventions — fine-tune. You can prompt your way to 80% compliance; the last 20% costs ten times the prompt length and breaks at scale. Cursor, the AI code editor, fine-tunes models for in-house tab completion because the format is rigid and the latency budget is single-digit milliseconds.

Narrow domain expertise. Medical coding, legal contract drafting, chemical synthesis — domains with their own vocabulary and patterns of reasoning that the base model has seen but does not reflexively use. Fine-tuning bakes the patterns into the weights so the model defaults to them rather than producing generic prose dressed up in jargon.

Latency and cost at scale. A fine-tuned 8B-parameter Llama can match a much larger general model on a narrow task while running ten times cheaper and faster. If you are processing millions of calls a day, this is the difference between a viable product and a venture-funded science project. OpenAI's own GPT-4o-mini, fine-tuned on the right data, often beats GPT-4 on specific workflows at a fraction of the price.

The tradeoff: every base-model upgrade resets your work. When the next generation arrives, your current fine-tune is suddenly the second-best option. You also need clean training data — a few thousand high-quality examples, minimum — and most teams do not have it. Fine-tuning on bad data is faster than waiting; it is also worse.

When long context wins, and how to actually choose

Long context is the right answer surprisingly often, and it is getting more often as windows grow and prices fall.

One-shot tasks. "Read this 80-page contract and flag the unusual clauses." "Summarise this earnings call." "Compare these two job descriptions." There is no corpus to manage, no behaviour to change — just a single query against a single chunk of text. Building RAG for this is engineering theatre.

Small, stable corpora. A startup with 200 internal docs, total. A founder's email archive. A codebase under 200K tokens. Claude and Gemini can hold the whole thing in context now. The marginal effort of building retrieval infrastructure for a corpus that fits in the prompt is mostly procrastination disguised as architecture.

Prototyping. You can validate whether AI solves your problem at all by stuffing the docs into a prompt in an afternoon. If it works, then decide whether RAG or fine-tuning is the production answer. If it does not work in long context, it almost certainly will not work in RAG either — RAG is a way to scale a working solution, not to rescue a broken one.

The tradeoff: cost and lost-in-the-middle. At 200K tokens of GPT-4 input, you are paying roughly $2 per query. Anthropic's prompt caching, generally available since late 2024, cuts this around 90% if the documents repeat — but only for repeat queries. And every long-context model still degrades on "needle in a haystack" recall: stuff a critical sentence at token 350,000 of a Gemini call and the model finds it 90% of the time, not 100%, which is fine for a chatbot and unacceptable for contract review.

So, the decision tree, in three questions, in order. One: does your corpus fit in context? If yes, and the use case is one-shot or low-volume, stop. Use long context. You will save weeks of engineering. If the corpus is small and stable but volume is high, use long context with prompt caching — same answer, 90% cheaper after the first call.

Two: does the knowledge change, or do you need citations? If yes, you need RAG. Period. Fine-tuning cannot give you fresh knowledge or sources, and stuffing 50,000 documents in context every query is economically insane. Build retrieval, accept the latency, and budget six months of "improving retrieval quality" because that is how long it actually takes.

Three: do you need to change how the model behaves, format outputs, or run cheap and fast at scale? If yes, fine-tune. Often on top of RAG — these are not mutually exclusive. The most sophisticated production systems (Harvey for law, Hippocratic for medicine) fine-tune a base model on domain reasoning, then use RAG to feed it fresh case data. Best of both, twice the engineering.

The named tradeoffs, on one line each. Cost: long context highest per query, RAG middle, fine-tune lowest at scale. Latency: long context worst, fine-tune best, RAG in between. Freshness: RAG wins, long context fine if you re-paste, fine-tune is frozen the day you ship it. Accuracy on your data: fine-tune highest if data is clean, RAG highest if data is large, long context highest for one-shot.

What most people miss: these are not ranked in difficulty. Long context is the easiest to start, but the cost curve is brutal. Fine-tuning is the hardest to do well, but the cheapest to run. RAG looks middle-ground but has the longest tail of edge cases. Pick the one that matches your problem, not the one that signals technical sophistication.

What this means for you

If you are a builder, the practical path is almost always: prototype in long context, move to RAG when the corpus grows or freshness matters, fine-tune only when you have hit the limits of prompting and you have clean training data. Doing it in any other order — fine-tuning before you have validated the use case in a prompt, building RAG before you have checked whether the corpus fits — is how teams burn three months and end up at the same answer.

If you are an investor, the interesting question is which of these three commoditises first. Long context is already racing to the bottom — Gemini, Claude, and GPT cluster around the same prices and capabilities. Fine-tuning infrastructure (Together, Fireworks, OpenAI's own service) is consolidating fast. RAG remains the messy middle, and the companies winning there — Glean, Perplexity, Hebbia — are winning on retrieval quality and product UX, not on model access. The model layer is becoming a utility; the retrieval and orchestration layers are where margin lives.

If you are a buyer evaluating an AI vendor, ask one question: "what happens when our knowledge base updates?" If the answer involves retraining, run. If the answer is "we re-index, takes about 20 minutes," they have built it correctly. The architecture choice tells you whether the vendor understood the problem or just shipped a demo.

Ask Gradient

This guide gave you the framework. The hard part is applying it to your specific case — which corpus, which budget, which latency target. Gradient's Ask Gradient RAG product is built on the same architecture this guide describes, indexed across our full library of AI research and frameworks. Try a question like "Should a 50-person legal SaaS with two million pages of case law use RAG, fine-tuning, or both?" and watch the decision tree get applied in real time, with sources you can verify. Related reading: What is an LLM, really? and Why context windows matter (and why they don't).