What Is a Transformer? The 2017 Architecture Behind Every LLM

Eight Google researchers, one paper, every LLM since

In June 2017, eight researchers at Google Brain uploaded a paper titled "Attention Is All You Need" to arXiv. It collected polite citations for two years, then quietly became the architecture under GPT-4, Claude, Gemini, Llama, and almost every model anyone has heard of.

If you have nodded along when someone said "transformer" and hoped the conversation moved on, this is the explanation you actually wanted. No equations — just the idea, why it worked, and what it still cannot do.

Before 2017: the sequence problem

Language is hard for computers because words depend on other words, sometimes very far away. "The book that the woman who lives next door lent me last year is overdue" — to handle "is overdue," the machine needs to know it refers to "the book," twelve words back.

The pre-2017 answer was the recurrent neural network — RNN, a model that reads text one word at a time and tries to keep a running memory of what it has seen. Its more sophisticated cousin, the LSTM (long short-term memory, 1997), did the same with a smarter forgetting mechanism. Google Translate ran on LSTMs until 2016.

The problem was structural. RNNs read sequentially, like a person reading aloud, which meant you could not split the work across multiple processors. Worse, by the time the model reached word 100, the signal from word 1 had been compressed and re-compressed so many times it had effectively faded. Long-range dependencies — the very thing language is full of — were exactly what these models were worst at.

By 2016 the field had a clear bottleneck. A better model could not just be a deeper RNN, because depth multiplied training time linearly and the memory problem stayed.

Attention Is All You Need

The paper, by Ashish Vaswani and seven co-authors, proposed throwing recurrence out entirely. No reading word by word. Instead, every word in a sentence would look at every other word in parallel and decide which ones to pay attention to. The mechanism — called self-attention — was not new; previous models had used attention as a helper bolted onto an RNN. The paper's claim was that attention alone was enough.

It was originally a translation paper. The benchmark was English-to-German, and the transformer beat the prior state of the art while training faster. That was the headline contribution at NeurIPS 2017. Almost no one in the room realised they were looking at the architecture that would, five years later, be running ChatGPT.

The slow uptake is itself instructive. OpenAI's GPT-1 (2018) and Google's BERT (2018) were the first to scale the architecture beyond translation. GPT-2 (2019) hinted at general capability. GPT-3 (2020), with 175 billion parameters, was the moment the rest of the industry — and the rest of the world — understood that the 2017 paper had been the inflection point. The lag between the idea and the realisation is a useful reminder that you can be in the room when history is made and still miss it.

Self-attention, in one sentence

Every word looks at every other word, and asks: which of you are relevant to me right now?

Take the sentence: "The animal didn't cross the street because it was too tired." What does "it" refer to? A human reads "it" and instantly looks back at "the animal" — not "the street" — because tired things are usually animate. A transformer does the same. When processing "it," it computes a relevance score with every other word in the sentence, and the score for "animal" comes out highest. That score then weights how much "animal" contributes to the model's understanding of "it."

This happens for every word at once, in both directions, and at multiple levels — what the original paper called multi-head attention, meaning the model runs several attention computations in parallel, each looking for a different kind of relationship: subject-verb, pronoun-antecedent, modifier, and so on. The model is not told what to look for; it figures that out during training.

The crucial point is that the model is not following grammar rules anyone wrote down. It is learning, from billions of examples, which words tend to matter to which other words in which contexts. Grammar emerges as a side effect.

Why this won: parallelism and scale

The deeper reason transformers took over is not really about language. It is about hardware.

An RNN processes word 1, then word 2, then word 3 — strictly serial. A transformer processes all words at once, because the attention computation between any two words is independent of any other pair. That makes the workload embarrassingly parallel, which is exactly the kind of workload a GPU — graphics processing unit, originally designed to render millions of pixels in parallel for video games — was built for.

This unlocked scale. The Kaplan paper (2020, OpenAI) showed that as you increase parameters, training data, and compute, transformer performance improves smoothly and predictably. The Chinchilla paper (2022, DeepMind) refined the recipe: most large models had been undertrained on data, and you get more for your dollar by feeding a smaller model more tokens. Either way, the relationship is a curve you can put on a slide and forecast against. The "scaling hypothesis" — that you can largely keep throwing compute at the same architecture and get a better model — is the single most consequential idea in modern AI, and it is true because of the transformer's parallelism.

NVIDIA's data-center revenue was around $3 billion in fiscal 2020. By fiscal 2024 it was over $47 billion. That is the scaling hypothesis cashing out as one company's balance sheet.

What transformers still can't do

A transformer is, fundamentally, a very good next-token predictor. It outputs the most likely next word, given everything it has seen. Three things follow from this, and they explain most of the failure modes you encounter in the wild.

It does not reason in any rigorous sense. It pattern-matches. When GPT-4 solves a maths problem, it is producing the kind of text that maths solutions usually look like, which works astonishingly often and fails embarrassingly when the problem is novel. The field is patching this with chain-of-thought prompting (write your steps), tool use (call a calculator), and "reasoning models" like OpenAI's o1 and Anthropic's extended thinking, which essentially let the model run longer before answering.

It does not remember. Every conversation starts cold. The context window — the chunk of text the model can look at in one go — has grown from 2,000 tokens (GPT-3, 2020) to 1 million (Gemini 1.5, 2024), but a million tokens is still finite, and the model retains nothing after the session closes. The patch is retrieval — letting the model query an external database at runtime, which is the basis of every "chat with your documents" product, including the one at the bottom of this guide.

It is not deterministic. Ask the same question twice, get two different answers. This is partly a deliberate design choice — sampling from probabilities makes outputs less repetitive — but it makes transformers a poor fit for anything that needs an exact, repeatable result: accounting, regulated workflows, formal verification.

What most people miss is that none of these are bugs that a bigger model will fix. They are properties of the architecture. The patches — RAG (retrieval-augmented generation), tool use, agent frameworks, reasoning loops — are doing real work, and the next interesting wave of progress is happening there, not in raw parameter count.

What this means for you

If you are using these tools at work, the practical takeaway is that the model is brilliant at text it has seen variations of, and unreliable at anything truly outside its training. Treat its output the way you would treat a confident intern who reads fast and lies smoothly. Verify anything load-bearing.

If you are watching the industry, the question is no longer "does the architecture work" — it does — but "how much further does scale go, and what comes next." Mamba (2023) and other state-space models are the most credible challengers, but transformers have an enormous incumbency advantage in tooling, talent, and hardware. Bet on transformers for at least the next three years; keep an eye on the alternatives anyway.

If you are a citizen trying to make sense of the news, the useful frame is this: a transformer is a statistical model of language, trained on a large fraction of the internet. It does not know things; it has absorbed patterns of how things are said. That distinction explains both why it is so useful — most knowledge work is, in practice, recombining patterns — and why it occasionally invents a court case. Mata v. Avianca (2023) is the canonical example: a New York lawyer cited six fake cases generated by ChatGPT in a real federal filing, and got sanctioned.

Ask Gradient

If you want to go further — into how attention actually weights words mathematically, or how multi-head attention differs across the major model families — point Ask Gradient at the question. Try: "How does the attention mechanism in Claude 3 differ from the one in the original 2017 transformer paper?" The system retrieves from primary sources, including arXiv papers and lab technical reports, so the answer is grounded in what was actually published rather than what the model happens to remember. For the next step up, the companion guide What Is an LLM? covers what happens when you scale this architecture to hundreds of billions of parameters.