What Is an LLM? The Plain-English Guide With No Math, No Hype

It is, fundamentally, a machine that guesses the next word.

GPT-4 did not pass the bar exam because it understood law. It passed because, given a prompt, it correctly guessed what word came next, then the next, then the next — about a thousand times in a row, on a topic it had read enough of to fake competence convincingly.

This guide explains what a large language model actually is — mechanism, training, limits — without a single equation. By the end you will know what "GPT" stands for, why context windows matter, why DeepSeek shipping a competitive model for a fraction of OpenAI's training budget rattled the industry, and why these systems hallucinate even when they are being earnest. It will not make you a researcher. It will make you harder to fool.

Prediction, scaled until it stops looking like prediction

An LLM does one thing : it reads a sequence of text and predicts the next token — a token being a chunk of text, usually three or four characters, roughly three quarters of an English word. Type "the cat sat on the" and the model assigns a probability to every possible next token in its vocabulary. "mat" gets a high score. "Wittgenstein" gets a low one. It picks one — usually with a touch of randomness, controlled by a setting called temperature — then repeats the trick using its own output as the new input. That is the entire loop.

The "large" in large language model refers to two things : the amount of text the model trained on, and the number of parameters — internal weights, the tunable dials that store what the model learned. GPT-3 had 175 billion parameters in 2020. GPT-4 is widely estimated above a trillion. Llama 3.1 (Meta, 2024) shipped at 405 billion. The numbers matter, but not in the way the headlines suggest.

The architecture underneath all of this is the transformer — introduced by eight Google researchers in a 2017 paper titled "Attention Is All You Need." Every modern LLM — from ChatGPT (the GPT stands for Generative Pre-trained Transformer) to Claude to Gemini to the Mistral models that fit on a laptop — is a transformer. There has been no comparable architectural shift in eight years. The progress has come from scale, data quality, and training tricks, not from a new shape of network.

The interesting question is not how prediction works. It is why prediction, applied to enough text, starts looking like reasoning.

Pretraining reads the internet. Post-training makes it useful.

An LLM is built in two phases, and they could not be more different in spirit.

First comes pretraining. The model is fed an enormous corpus — Common Crawl (a public web scrape going back to 2008), books, code repositories, Wikipedia, academic papers, often a few trillion tokens in total. It is given one job : predict the next token. No labels, no instructions, no notion of helpfulness. Just billions of passes over the data, with the parameters nudged each time toward better predictions. This phase costs tens to hundreds of millions of dollars in GPU time. It is what turns a random network into something that knows what English looks like, what Python looks like, what a legal contract looks like, what a sonnet looks like.

The result of pretraining is what researchers call a "base model." A base model is not a chatbot. Ask one a question and it might continue your sentence with another question, or write the next ten questions in what it assumes is a quiz. It has read the internet, but it has not been told what humans want from it.

That is the job of post-training. The base model is fine-tuned on curated examples of useful answers — a process called instruction tuning — then refined further using human feedback : raters compare two responses, the model learns to prefer the one humans liked, a technique called RLHF (reinforcement learning from human feedback) introduced by OpenAI in 2022 and now standard everywhere. Anthropic uses a variant called Constitutional AI, where the model critiques itself against a written set of principles. The details vary; the goal is the same — turn a fluent autocomplete into something that follows instructions, declines to help with bioweapons, and admits when it does not know.

What most people miss is that nearly all the personality you experience — the tone, the caveats, the refusals, the helpfulness — lives in post-training. The intelligence is from pretraining. The character is from the lab.

Parameters, Chinchilla, and the death of "bigger is better"

For three years after GPT-3, the assumption in the field was that scaling parameters indefinitely would scale capability indefinitely. The Kaplan scaling laws (OpenAI, 2020) seemed to confirm it. Lab CEOs said as much in interviews.

Then DeepMind published Chinchilla in March 2022 and quietly rearranged the playing field. Their result : for a given compute budget, the optimal model is much smaller, trained on much more data, than anyone had been building. GPT-3 at 175B parameters had been trained on roughly 300 billion tokens. Chinchilla showed it should have seen ten times more. Most of the giant models of 2020-2022 were undertrained. Llama 3, released in 2024, took the lesson seriously — a 70B model trained on 15 trillion tokens that punches well above its weight.

The other escape from raw size is Mixture of Experts — an architecture in which the model contains many specialised sub-networks ("experts") and routes each token to only a few of them. The total parameter count can be huge, but the compute per token stays modest. Mistral's Mixtral, GPT-4 (widely believed to be a MoE), and DeepSeek-V3 all use this trick. DeepSeek-V3 has 671 billion total parameters but activates only 37 billion per token, which is part of how it shipped at a fraction of GPT-4's training cost in late 2024 and made every Western lab nervous.

Size still matters, but the honest version of the rule is : compute matters, training data matters, post-training matters, and the architectural choices around how you spend your compute matter more than the headline parameter count.

The context window is the model's working memory — and it is short

An LLM does not remember you between conversations. It does not remember earlier in the same conversation either, unless that earlier text is still inside its context window — the chunk of text it can see at once when generating the next token. Everything outside the window might as well not exist.

Context windows have grown fast. GPT-3 had 2,048 tokens, about 1,500 words. GPT-4 launched in 2023 at 8K, then 32K, then 128K. Claude 3 (Anthropic, 2024) shipped at 200K — roughly a 500-page book. Gemini 1.5 (Google, 2024) demonstrated 1 million tokens, occasionally 10 million in research settings. Practical effective recall over those long windows is still imperfect, but the trajectory is clear.

The window matters because everything the model "knows" about your specific situation has to fit inside it. Your previous messages, the document you uploaded, the system prompt the application built around it, the tools it has access to — all of it competes for space. When developers connect an LLM to a database of company documents using retrieval-augmented generation (RAG — fetch relevant text, paste it into the context window, then ask), the entire dance is about getting the right information into the window before the model answers. RAG is its own subject.

The current cast, and what actually differs

By 2025 the frontier is occupied by a handful of labs, and the differences between their flagship models are real but smaller than the marketing implies.

OpenAI ships GPT-4 and its successors. The reigning consumer brand, the largest deployment, generally strong at code and structured reasoning. Closed weights, API only.

Anthropic ships Claude 3 and its successors (Claude 3.5 Sonnet, Claude 3 Opus). Best-in-class at long-context comprehension and writing, with a noticeably different "voice" from GPT — drier, more cautious, less prone to confident nonsense in many evals. Closed weights.

Google DeepMind ships Gemini, the only frontier model trained natively on text, image, audio and video together (multimodal from the ground up). Strongest tie-in to Google products, longest context window in production.

Meta ships Llama 3 with open weights. You can download the parameters, run them on your own hardware, fine-tune them on your own data. This single decision has done more to seed the open-source ecosystem than any other corporate move. Mark Zuckerberg's stated reason : commoditise the layer Meta does not want anyone else to own.

Mistral, a French lab founded in 2023, ships Mistral Large (closed) and a stack of strong open-weight models. Smaller, faster, the European hedge against US dominance.

DeepSeek, a Chinese lab, shipped DeepSeek-V3 and DeepSeek-R1 in late 2024 and early 2025 — competitive with the US frontier at training costs reported around $6M, an order of magnitude below what Western labs claim to spend. Open weights. Whether that cost figure tells the whole story is debated; that the gap with Silicon Valley is narrower than assumed is not.

What differs across these models, in practice : the training data mix, the post-training recipe, the safety philosophy, the price per token, the licence. What does not differ much : the underlying architecture, the basic mechanism, the fact that all of them are, fundamentally, very expensive autocomplete with very good taste.

What this means for you

If you are using LLMs as a tool — to write, summarise, code, draft — the practical implication of all this is that the model is a confident pattern matcher with no grounding in truth. It will produce a fluent answer whether or not it has any idea what it is talking about. Hallucinations are not a bug being fixed in the next version. They are a property of the mechanism. Treat every factual claim as a hypothesis to verify, especially in domains the model has read little about (your specific company, recent events, niche legal questions).

If you are choosing between models, ignore the leaderboards and test on your actual task. The differences between GPT-4, Claude 3 and Gemini at the top tier are real but task-dependent. For long documents, Claude. For tightly integrated tooling, GPT. For multimodal, Gemini. For running locally or fine-tuning, Llama or Mistral. For cheap inference at scale, the open models or DeepSeek.

If you are watching as a citizen or investor, the question that matters is no longer "can the models do impressive things." They can. The question is whether the next generation continues to scale economically — whether GPT-5 and Claude 4 deliver returns proportional to their training cost, or whether the field hits a wall and the action moves to specialised models, agents and applications. Chinchilla and DeepSeek both argue the wall is already visible.

Ask Gradient

For deeper questions about a specific model, training method or capability, our Ask Gradient RAG tool searches our full library of explainers, papers and analysis. Try asking it : "How does Claude 3's Constitutional AI training actually differ from RLHF?" — or whichever question this guide left you wanting to ask.