The Scaling Hypothesis: Why It Worked and Where It's Breaking

A $30 billion bet on a single straight line

In 2020, OpenAI published a paper arguing that the smartness of a language model — the kind that writes essays and code — was a function of three numbers : compute, parameters, and data. Plot them on a log-log chart, where each gridline represents a tenfold step, and the points fell on a straight line.

That line is the most expensive prediction ever made in technology. Every AI investment since rests on the bet that it would keep going. This guide is about why it paid out for four years, where it's wobbling, and what's quietly replacing it.

What Kaplan actually found

The paper — "Scaling Laws for Neural Language Models", Kaplan et al., January 2020 — trained language models across many sizes and watched how their loss curves moved. Loss is the model's error : how surprised it is by the next word in a sentence, on average. Smaller loss, smarter model.

Kaplan's team found loss dropped predictably as you grew compute (the FLOPs — floating-point operations — burned during training), parameters (the dials inside the network), or data (tokens — roughly, fragments of words — fed to it). Double any one of them, loss drops by a fixed amount. No diminishing returns in sight, at least up to the scale they had tested.

This was not theory. It was empirical : the line just kept being a line. The implication was awkward and intoxicating. If smartness was a continuous variable controlled by spending, then GPT-4 was a checkbook problem.

GPT-3 and the year everyone believed

OpenAI ran the bet six months later. GPT-3 — June 2020, 175 billion parameters, several million dollars of compute — was the first model that could be prompted instead of fine-tuned. Show it three examples of translation and it translated. Ask it to write Python and it wrote Python. No one had told it to.

This was the moment the scaling hypothesis stopped being a plot in a paper and became an industry. Microsoft put $1B into OpenAI in 2019 and another $10B in 2023. Google, Meta, and Anthropic followed. NVIDIA's data-center revenue, around $3B per quarter in 2022, was past $30B per quarter by late 2024. All of it on the conviction that the line would keep being a line.

What most people miss : the bet was not that bigger models would be more useful. It was that they would behave qualitatively differently. Translation, arithmetic, basic code — none of these were specifically trained for. They emerged, abruptly, around certain scale thresholds. Anthropic's CEO Dario Amodei has called this "the hypothesis that intelligence is a continuous quantity you can buy." That sentence either makes you a believer or makes you angry.

Chinchilla : the correction nobody priced in

In March 2022, DeepMind published "Training Compute-Optimal Large Language Models" — the Chinchilla paper. They argued the industry had been doing it wrong. Kaplan had mostly varied parameters; he had not varied data nearly as aggressively. When DeepMind did, the optimal recipe shifted.

The headline result : for a given budget, you want roughly 20 tokens of training data per parameter. GPT-3 had under two. It was, in that sense, a wildly inefficient model — a 175B-parameter brain fed too few books. Chinchilla itself was 70B parameters trained on 1.4 trillion tokens. It beat GPT-3 across the board on a smaller compute budget.

The correction reframed the bet. The scaling hypothesis still held — but the scarce resource wasn't GPUs. It was high-quality text. Every lab quietly redirected its scraping operations. By 2024, frontier models like Llama 3 were training on around 15 trillion tokens. That number is close to the entire useful public web.

Where the line is breaking

Three problems are now visible at the frontier.

First, data exhaustion. The stock of high-quality English text — books, papers, well-edited web — is finite, and the frontier labs are inside the asymptote. Epoch AI, a research group that tracks compute trends, estimated in 2024 that quality web text would run out for training somewhere between 2026 and 2028. Synthetic data — text generated by another model — helps, but it carries its own contamination risk : a model trained on its own outputs tends to drift.

Second, diminishing returns. GPT-4 (March 2023) felt like a step change over GPT-3.5. The models since — GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 405B — feel like sidegrades. They are cheaper, faster, more multimodal, but the raw "this thing is alarmingly smarter than the last one" sensation has thinned. Sam Altman has said GPT-5 is coming. He has not said it will be the leap GPT-4 was.

Third, inference cost. A bigger model costs more every time someone uses it, forever. ChatGPT serves north of 200 million weekly users. At GPT-4 scale, the marginal cost of intelligence is no longer a footnote. The economic ceiling on parameter count is now lower than the technical one — labs could build a 10-trillion-parameter model; nobody could afford to query it.

What replaced pure scaling

By late 2024, frontier labs had quietly stopped racing on parameter count and started racing on three other things.

The first is post-training : taking a finished base model and teaching it to be useful. RLHF — reinforcement learning from human feedback, the technique behind ChatGPT's politeness — was the first version. DPO (direct preference optimization, Stanford 2023) made it cheaper. A well-post-trained 70B model now beats a poorly post-trained 400B one on most user-facing tasks. The dials moved inward, from raw scale to refinement.

The second is tool use. A model that can call a Python interpreter, a search engine, or a database does not need to memorize the world's facts inside its weights. Anthropic's Claude, OpenAI's GPT-4, and Google's Gemini all became agentic during 2024 — meaning they take actions, not just produce text. This trades scale for plumbing, and plumbing is much cheaper.

The third, and most interesting, is test-time compute. OpenAI's o1 (September 2024) and o3 (announced December 2024) do something different : they think for longer before answering, generating long internal chains of reasoning the user never sees. Instead of paying for a bigger model once at training time, you pay for a longer thought process each time at query time. On math and coding benchmarks, o3 jumped past every prior model on tasks the field had been stuck on for two years. The new scaling axis isn't training compute. It's inference compute.

The interesting question isn't whether scaling is dead. It isn't. The line still works for the next few orders of magnitude, and Meta, Google and xAI are all still building 100,000-GPU clusters. The question is whether the next leap looks like GPT-3 → GPT-4 (a bigger model) or GPT-4 → o3 (a smarter algorithm running for longer on a similar-sized model). The smart money is splitting bets.

What this means for you

If you build with these models, the practical lesson is to stop assuming the next generation will solve your problem for free. From 2020 to 2023 you could ship a half-broken prompt and wait six months for the next model to fix it. That arbitrage is closing. Engineering effort — better prompts, retrieval, tool use, evaluation harnesses — is now where the gains are.

If you invest, watch inference economics, not parameter counts. The question to ask any AI startup is not "what model are you using" but "what does it cost you per query, and how does that change at 10× volume." NVIDIA's growth from here depends more on inference workloads than on training runs. So does every model lab's margin.

If you are simply trying to make sense of the headlines, here is the frame : the era of free progress from spending more is ending. The era of progress from cleverer architectures, longer reasoning, and better feedback loops has started. It is slower, less photogenic, and probably more important. Anyone telling you AGI arrives in 2026 because the line goes up forever has not read the last six months of papers.

Ask Gradient

Want to dig further? Our Ask Gradient — a RAG (retrieval-augmented generation) system that answers from a curated library of papers, transcripts, and earnings calls instead of a model's frozen memory — has every source referenced above indexed. A good starter question : "How much did training compute grow per year between GPT-2 and GPT-4, and what does Epoch AI project for 2026?" You'll get a sourced answer with the papers attached. If you want the underlying machinery first, our guide to what an LLM actually is is the natural next stop.