How to Read an LLM Benchmark — and Why Most of Them Are Gamed

The day a held-out test exposed half the leaderboard

In May 2024, Scale AI built GSM1K — a fresh, held-out twin of the famous grade-school math benchmark GSM8K — and re-ran the leaderboard on it. Several widely-cited small models dropped up to thirteen points. Same arithmetic, different paper, different score.

This guide is the toolkit for not being fooled by the next chart. By the end you'll know what the famous benchmarks — MMLU, GPQA, HumanEval, SWE-Bench, ARC-AGI — actually measure, the four ways they get cooked, and the cheap, paranoid moves real evaluators make instead.

What a benchmark actually is (and why that's fragile)

An LLM benchmark — a frozen test set evaluated on a frozen model — is a stack of questions with known answers. You feed each question to the model, grade the response with an automated rule (multiple-choice match, exact string, unit tests passing), and divide. That arithmetic is what the entire industry's leaderboards rest on.

The setup borrows from psychometrics, where the test is hidden and the test-takers don't get to study the answer key. LLM benchmarks reverse that. The questions are public, the answers are public, and the test-takers are trained on roughly the entire internet — which is where the questions and answers live. The benchmark is a closed-book exam being written by students who've already photocopied the textbook.

This isn't fraud. It's structural. The same property — train on every text — that lets these models work also makes the test/train boundary almost impossible to keep clean. Any benchmark older than the model has probably leaked. For a refresher on what training even is, see our guide on what an LLM actually is.

The big five — what each one is actually for

MMLU (Hendrycks et al., 2020) — 57 multiple-choice subjects from US bar exam to college physics, ~14,000 questions. It was the dominant general-knowledge test from 2021 to 2024. By 2024, GPT-4 was around 86%, Claude 3 Opus around 87%, and the ceiling had become the noise floor. MMLU is now mostly a sanity check.

GPQA (Rein et al., 2023) — 448 graduate-level biology, physics, and chemistry questions written by domain PhDs and designed to be "Google-proof." Domain experts score around 65% with 30+ minutes and full internet access. GPT-4 launched near 35%; by mid-2024 frontier models reached the high 50s. Harder to contaminate because it's small and was carefully gated. It will saturate too.

HumanEval (Chen et al., 2021, OpenAI) — 164 hand-written Python problems graded by unit tests. Useful through 2022, saturated by 2023. Today, a model under 90% on HumanEval is not a serious coding model — and a model at 95% may still be useless on your repo.

SWE-Bench (Princeton, 2023) — 2,294 real GitHub issues from twelve popular Python projects (Django, sympy, scikit-learn) that the model must actually patch. Grading uses the project's own test suite. SWE-Bench Verified, the human-curated subset of 500 issues, is the closest thing the field has to honest work, which is why it's the benchmark Anthropic, OpenAI, and Cognition all quote. Frontier scores climbed from ~5% in 2023 to roughly 50% by late 2024 — real progress, on a real task.

ARC-AGI (Chollet, 2019) — small grid puzzles where a few input/output examples imply a transformation rule the model must apply to a new grid. Designed to resist memorization because the puzzle space is novel. Pure language models stalled near 5%. OpenAI's o3, in December 2024, hit 87.5% on the semi-private set — at a reported cost of around $3,400 of compute per task on the high-compute setting. ARC-AGI is the benchmark that most aggressively separates pattern-matching from reasoning, which is why you should care about it even though no one ships it.

The interesting question isn't which model wins each. It's what each one tests. MMLU tests recall under multiple choice. GPQA tests recall plus reasoning under multiple choice. HumanEval tests if you can write a function from a docstring. SWE-Bench tests if you can fix a real bug in a real codebase. ARC-AGI tests if you can do something genuinely new. These are wildly different cognitive tasks, and the same model can be world-class on one and embarrassing on another.

The four ways benchmarks get gamed

1. Contamination. The training corpus contains the test set. Sometimes by accident — a Common Crawl scrape happened to include a paper quoting MMLU questions. Sometimes by sloppy curation — a fine-tune dataset full of problems that look suspiciously like GSM8K. Scale AI's GSM1K work (Zhang et al., 2024) re-tested leading models on a held-out twin and watched several small open models drop double digits. GPT-4, Claude, and Gemini barely moved. Translation: the small models had memorized; the frontier models had generalized — at least on grade-school math.

2. Narrow optimization. If a benchmark is famous enough, someone will reinforcement-learn specifically against it. The model becomes excellent at the format and no better at the underlying skill. This is why a 95% HumanEval doesn't mean your agent will refactor your auth module — it means the model has been fed a lot of "write a Python function from a docstring" feedback.

3. Cherry-picking. Every launch post includes a comparison table. The columns are picked by the launching lab. They show the benchmarks the new model wins. They quietly drop the ones it loses. Read enough launch posts and you notice that no two labs ever pick exactly the same six benchmarks. That is not a coincidence.

4. Inference-time inflation. The same model with chain-of-thought prompting, majority-voting over 64 samples, or tool use can score 20 points higher than its raw single-shot output. This is fine — it's how models are actually deployed — but a cell that reads "73%" can hide thousands of dollars per query of difference. The o3 ARC-AGI headline number is a useful research result and a useless product spec at the same time.

None of these are conspiracies. They are the rational responses of labs in a scoreboard race that the press treats as gospel. Labs publish in good faith; they publish in their own favor. What most people miss is that "84.2 on MMLU" is a marketing claim wearing a number's clothes.

How to actually evaluate a model

The professional move is to assume every public benchmark is partially compromised and look for two things instead: held-out evaluations and your own tasks.

Held-out evals keep the questions private. ARC-AGI has a private set Chollet uses for the actual prize. SWE-Bench Verified rotates issues. LiveBench (Abacus AI, 2024) refreshes its questions monthly so they can't have been in pretraining. The Aider polyglot leaderboard runs a fresh suite of multi-language code edits and is brutal to models that looked great on HumanEval. Chatbot Arena (lmarena.ai) does pairwise human preference voting at scale — gameable in different ways, but the test set is "whatever 100,000 humans actually wanted to ask," which is not in any corpus.

The amateur move is to read the leaderboard. The professional move is to write fifty prompts that look like your actual problem, run them on three or four candidate models, grade the outputs yourself or with a stronger model as judge, and pick the winner. This sounds laborious. It takes an afternoon. It is the difference between deploying a model that handles your support tickets and deploying one that handles MMLU.

What this means for you

If you're a builder choosing a model, don't pick by benchmark. Pick by an evaluation you wrote against your own data, last week. Anthropic, OpenAI, and Google all publish small evaluation harnesses; you want maybe 30 to 100 representative examples with a grading rubric. The cost is one engineering afternoon and the saving is months of debugging the wrong model.

If you're an investor watching a launch slide, the question isn't "did this model beat GPT-4 on MMLU." Everything beats GPT-4 on MMLU now. The question is: which benchmarks did they not show? Did they include SWE-Bench Verified, LiveBench, or Aider — the ones that resist contamination? Did they report inference-time compute? A model that wins benchmarks at $40 per query is a research result, not a product.

If you're a citizen reading an "AI achieves human-level performance on X" headline, translate it as: "a model scored above the human average on a frozen test set of X-shaped questions, on the day a researcher wrote a press release." That is sometimes a real achievement. It is rarely the thing the headline claims. The most reliable signal that a benchmark matters is when its scores stop going up — meaning the test is finally hard enough that the easy gaming is exhausted.

You don't need to memorize benchmark names. You need one habit: when someone shows you a number, ask what test, what conditions, what compute, and what's missing.

Ask Gradient

If you want to go deeper on how a specific lab evaluates its model — for example, "how did Anthropic test Claude 3.5 Sonnet on coding, and what contamination checks did they run?" — put the question to Ask Gradient, our retrieval-augmented research assistant. It pulls from system cards, papers, and lab blog posts so you can read what was actually measured, not what the launch graphic showed. For the underlying technology, see also our companion guide on what an LLM actually is.