Superintelligence: Myth, Probability, and the Timeline Question

Geoffrey Hinton quit Google in May 2023 because he thought he'd helped build something dangerous

He had spent forty years on the neural network ideas underlying modern AI, won a Turing Award for it in 2018, and at age 75 walked out so he could say in public that the systems were getting smarter faster than he expected. He did not say superintelligence was here. He said he could no longer rule out that it was coming.

This guide is for the reader who has heard "AGI by 2027" from one camp and "it's all a bubble" from the other and would like a sharper mental model than either. We'll define the term, walk through the strongest version of each side, and end with a Bayesian frame — assigning probability across timelines — that lets you update as the evidence comes in instead of having to pick a tribe.

What "superintelligence" actually means

The term was sharpened by Nick Bostrom — Oxford philosopher, author of Superintelligence: Paths, Dangers, Strategies (2014) — to describe a system that outperforms the best human minds across essentially every domain that matters: scientific creativity, general wisdom, social skills, strategic planning. Not a faster calculator. Not a better chess engine. Smarter than the smartest humans at the things humans are smart at.

Distinguish this from AGI — artificial general intelligence — which usually means roughly human-level across most cognitive tasks. AGI is the floor. Superintelligence (often abbreviated ASI) is whatever sits above it. Bostrom proposed three flavors: speed superintelligence (a human-level mind running a thousand times faster), collective (millions of human-level minds in coordination), and quality (a mind qualitatively smarter than ours, the way ours is smarter than a chimp's).

The interesting question is not whether the labels are clean — they aren't — but whether the thing they point at is physically possible. On that, almost no serious researcher says no. The disagreement is entirely about timing and approach.

The case for soon: scaling, compute, and recursive self-improvement

The most concrete recent statement of the bull case is Leopold Aschenbrenner's "Situational Awareness" — a roughly 165-page essay published in June 2024 by the former OpenAI superalignment researcher — which argues, in short, that AGI arrives around 2027 and superintelligence shortly after. The argument rests on three pillars.

The first is scaling. The Kaplan paper from OpenAI (2020) and the Chinchilla paper from DeepMind (2022) showed that LLM — large language model — performance improves predictably with more compute, more data, and more parameters. Every order of magnitude of compute has, so far, bought roughly the same step up in capability. GPT-2 (2019) wrote incoherent paragraphs. GPT-4 (2023) passes the bar exam. The scaling hypothesis says: keep going.

The second is compute. Epoch AI tracks training-run compute and estimates it has been doubling roughly every six months for frontier systems — much faster than Moore's Law. NVIDIA reported over $100B in data-center revenue in fiscal 2025, the bulk of it AI training chips. Frontier labs are building $100B-class clusters. The physical infrastructure for several more orders of magnitude is already under construction.

The third is recursive self-improvement — the idea that once AI is smart enough to do AI research, the curve goes vertical. Ilya Sutskever, OpenAI co-founder, has framed this for years: the moment models can meaningfully accelerate model development, you stop measuring progress in years and start measuring it in months. Aschenbrenner's whole argument is that the "drop-in remote worker" version of this is closer than people think.

The bull case is not "the press releases are right." It's: the curve has been steady for fifteen years, the inputs are still scaling, and there's no specific reason to believe the next decade looks different from the last.

The case against: hallucinations are not a bug, they're the whole story

The bear case is held by people who have been in the field just as long. Gary Marcus — NYU emeritus professor, author of Rebooting AI (2019) — has argued for two decades that pure neural networks lack the symbolic-reasoning machinery needed for genuine generalization, and that scaling will hit a wall. Melanie Mitchell — Santa Fe Institute, author of Artificial Intelligence: A Guide for Thinking Humans (2019) — has documented in detail how current models fail at the kind of fluid analogy and abstraction a four-year-old handles trivially. Yann LeCun, Meta's chief AI scientist, says explicitly that LLMs as currently designed are an off-ramp, not a path to human-level intelligence; he has spent the last several years pushing JEPA — Joint Embedding Predictive Architecture — as an alternative.

Their core argument: hallucinations are not a polish problem. When GPT-4 confidently cited court cases that didn't exist — this happened to lawyer Steven Schwartz in Mata v. Avianca (2023), who was sanctioned for filing the brief — it wasn't because the model had incomplete knowledge of caselaw. It was because the model had no model. It was producing plausible token sequences. There is no internal representation of "true" versus "false" being consulted. Polishing that surface doesn't get you to a mind.

The benchmark that crystallizes this is ARC-AGI, designed by François Chollet (Google researcher, author of On the Measure of Intelligence, 2019). Humans solve roughly 80% of ARC tasks. GPT-4 scored around 5% in 2023. OpenAI's o3 cracked the benchmark in late 2024 — but at a reported cost of around $20 per task on the standard setting and orders of magnitude more on the high-compute one. The bears note: a system that needs $20 of inference to do what a human child does for free is not on a smooth glide path to godlike intelligence.

What most people miss is that the bear case is empirical, not metaphysical. It isn't "machines can never think." It's "this specific architecture, scaled this specific way, has specific failures that aren't going away."

What both camps agree on

Nothing currently shipping is superintelligent. Claude, GPT, Gemini, Llama — all of them — are extraordinarily capable in narrow ways and brittle in obvious ones. They cannot run a small business unsupervised for a month. They cannot do original mathematics at the level of a working researcher. They cannot maintain a coherent goal over weeks. The bulls and bears both know this. The disagreement is purely about what happens between here and the limit.

Both camps also agree the question is empirical, not philosophical. Nobody serious is arguing about consciousness or qualia or whether silicon can "really" think. The question is whether, given the architectures and compute we have or can build, you can get to a system that out-researches the best human researchers. That is a question about benchmarks and curves, not about souls.

How to actually think about it: the Bayesian frame

The mistake most readers make is treating this like a binary forecast — "AGI in 2027" versus "never" — and then picking a side based on tribal affiliation. The right move is probabilistic. You spread your belief across timelines and update as evidence arrives.

A reasonable starting distribution, roughly calibrated against current expert surveys and prediction markets like Metaculus and Manifold, might look something like: 10–15% probability of superintelligence by 2030, 30–40% by 2040, 50–60% by 2050, with the remainder scattered across "much later" and "not on this paradigm." These numbers are not the truth. They are a place to start. The point is to have a distribution, not a date.

Then you watch for signals that should move the numbers. Signals that should shift mass earlier: a frontier model that meaningfully accelerates AI research itself; sustained scaling progress past GPT-5-class systems; a new architecture that solves the abstraction problems Mitchell and Chollet have documented. Signals that should shift mass later: a clear plateau in benchmark improvements despite compute scaling; persistent hallucination rates that don't fall with size; capital pulling back as returns on training runs flatten.

What you don't do is pick a year and defend it. The honest position, in 2026, is wide error bars and active updating.

What this means for you

If you build things — products, companies, careers — the operative question is not "when does ASI arrive?" It's "how do I stay valuable across the plausible range of outcomes?" Build skills and assets that compound regardless of timeline: judgment, taste, distribution, relationships, the ability to ship. These are robust to fast and slow worlds. Don't bet your decade on a 2027 date you cannot verify.

If you invest, watch the inputs, not the marketing. Compute spend, training-run scale, benchmark progress on hard tasks like ARC-AGI and FrontierMath, capital flows in and out of NVIDIA and the frontier labs. The narrative will lag the curve in both directions — bull markets overshoot, AI winters arrive after the data has already turned.

If you're a citizen — not a builder, not an investor, just someone who'd like to vote and live thoughtfully — the obligation is to refuse the binary. Don't outsource the question to the loudest person on your timeline. Read Aschenbrenner. Read Marcus. Read Mitchell. Hold both possibilities in your head at once. Superintelligence may be the most consequential open question of our century. It deserves more than tribal certainty.

Ask Gradient

If you want to go deeper on any thread in this guide — the scaling hypothesis, the alignment problem, what specifically Aschenbrenner gets right and wrong — try our Ask Gradient RAG tool. It pulls from current research and primary sources, not press releases. A good question to start with: "What would have to be true about current LLM architectures for the bear case to be wrong?" The answer is more interesting than either camp tends to admit. For more on the underlying technology, see our guides on what an LLM actually is and the alignment problem.