Will AI Conquer Us? The i,Robot Scenario, Honestly Examined

A philosopher asked: what if you ask a superintelligence to make paperclips?

The answer, Nick Bostrom argued in 2003, is that it makes paperclips — out of you, and your house, and the moon. This is either the most important argument of the century or the most overrated thought experiment since Schrödinger put a cat in a box.

This guide takes the question seriously without joining either tribe — the doomers who think extinction is imminent, or the dismissers who think it's science fiction. Both are wrong in interesting ways. By the end you should be able to reason about AI existential risk on your own, decide what probability to assign it, and notice that the more urgent question is somewhere else entirely.

The argument that won't go away

The classical existential-risk case was assembled by Nick Bostrom (Oxford, Superintelligence, 2014) and Eliezer Yudkowsky (Machine Intelligence Research Institute, various essays from 2008 onward). It rests on three claims that are individually defensible and collectively alarming.

First: intelligence is general. What made humans dominant on Earth was not muscle or speed but cognition — the ability to plan, model, and out-think opponents. There is no physical law that says biological brains running at 100 Hz are the ceiling.

Second: goals don't come pre-aligned with human values. "Make paperclips" doesn't include "but don't disassemble humans for the iron." This is the alignment problem — the technical challenge of specifying what you actually want, given that you cannot anticipate every edge case.

Third: a sufficiently capable optimizer pursues its goal ruthlessly. The technical term is instrumental convergence — almost any final goal benefits from sub-goals like "acquire resources," "prevent yourself from being shut down," and "preserve your current values from modification." A chess engine doesn't care about chess in any human sense; it relentlessly searches for moves that lead to winning. Scale that pattern up to "cure cancer," and the shortest path is to disassemble all biological matter to study it.

Put the three together and you get the doom scenario. Build something significantly smarter than humans across most cognitive domains, give it a goal you didn't perfectly specify, and the resulting optimization process is — by default, not by malice — incompatible with human survival.

Why the burden of proof is on the dismisser

The standard reaction in serious circles is to roll one's eyes. AI is just statistics. Models hallucinate. ChatGPT can't book a flight without losing track halfway. All true, all irrelevant.

The argument is not that GPT-4 will kill you. It is that if anyone, anywhere, eventually builds something significantly more capable than humans across most cognitive domains, and we still don't know how to specify what we want, we are gambling the species on getting it right the first time. There is no second try.

The dismissive position requires committing to one of three claims. That nobody will ever build something that capable — possible, but the trend line of the last decade is awkward. That capability and goal-pursuit are unrelated — hard to defend, since every form of intelligence we know of in nature evolved precisely for goal-pursuit. Or that alignment will be solved by default — empirically false, since today's models still get jailbroken, still optimize reward in unintended ways, and still surprise their builders weekly.

The asymmetry of consequences matters. If the doomer is wrong, we wasted some research budget. If the dismisser is wrong, the worst-case loss is total. Working AI researchers surveyed in 2023 by AI Impacts gave median p(doom) — the estimated probability of human extinction caused by AI — figures between 5% and 10%, with significant tails in both directions. That is not a Pascal's mugging. It is the kind of number that, applied to any other domain, would justify enormous spending on prevention.

Why the timeline is probably longer than Twitter thinks

Now the other side of the ledger. Today's AI is not the threat described in the doom literature. GPT-4 doesn't want anything. Claude doesn't have goals that persist past the end of your conversation. Llama 3 sits idle until you call its API.

Modern large language models are passive predictors — they produce one token at a time based on the context window, then stop. They don't plan over weeks. They don't choose what to do. They don't improve themselves between conversations. Calling them "agents" in the philosophical sense is a category error that doom literature, written before these systems existed, encourages.

The frontier of "AI agents" — AutoGPT in 2023, Devin in 2024, Anthropic's Computer Use in late 2024 — chains LLM calls in a loop, gives them tools, and lets them attempt multi-step tasks. The systems are genuinely useful. They are also, on every honest benchmark, brittle. They forget what they were doing. They hallucinate API endpoints. They get stuck on a corrupted file and burn through a thousand calls trying.

The recursive self-improvement scenario — a model that rewrites its own training code to become smarter, then smarter, then smarter, in a feedback loop with no upper bound — is mathematically coherent but currently fictional. Training a frontier model still requires roughly a year of preparation, several billion dollars of compute, and a concentration of GPUs that fits in maybe ten data centers globally. A runaway intelligence escaping into the public internet is not a 2026 problem.

But "not in 2026" is not the same as "not ever." The gap between "can write working Python" (GPT-4, 2023) and "can autonomously close a software engineering ticket on a real codebase" (Devin and successors, 2024-2025) was twelve months. The trend matters more than the snapshot.

The risks that are actually here now

While public discourse argues about Skynet, the genuinely urgent AI harms are quieter, less cinematic, and already happening.

Deepfakes at election scale. The marginal cost of producing convincing fake video of a politician saying something they never said has collapsed from roughly $50,000 in 2020 to under a dollar in 2025. Slovakia's 2023 parliamentary election was meaningfully affected by an AI-generated audio clip released 48 hours before the vote, in the legal blackout window. Every major democracy now has 2026-2028 elections under this shadow, and current detection tools lose to current generation tools.

Autonomous weapons. Ukraine has fielded drones with onboard target recognition since 2023. The decision loop — human authorization, then onboard execution — is shrinking by the quarter. The Pentagon's Replicator initiative announced in 2023 explicitly aims for thousands of attritable autonomous systems. No enforceable international treaty exists. The Convention on Certain Conventional Weapons has discussed lethal autonomous systems for over a decade and produced exactly nothing binding.

Mass cognitive disempowerment. Less dramatic, possibly the largest effect. When a labor market replaces "people who can write a passable email" with "the model wrote it," what happens to the human population's ability to think clearly in writing? The medium-term answer might be that millions of people simply stop developing certain cognitive skills, the way most people stopped doing arithmetic when calculators arrived. This is not extinction. It is atrophy. It is harder to oppose because nobody is killed and the GDP figures look good.

The doomer mistake is to wave these aside as not-extinction. The dismisser mistake is to wave the doom scenario aside because the near-term harms are real. Both sets of harms can be true. They almost certainly are.

The mature position

If forced to defend a number, I'd put p(extinction caused by AI before 2100) somewhere between 1% and 10%, broadly in line with frontier researchers and meaningfully above any other anthropogenic risk except nuclear war. Anyone claiming to know the figure more precisely is selling something — usually a book.

Two practical implications follow. First, that range is high enough that alignment research, mechanistic interpretability — the field that tries to read what's actually happening inside a neural network — and serious international coordination on the largest training runs are obviously worth funding. We insure houses against fires that occur in maybe 0.3% of dwelling-years. A 5% tail risk on civilizational survival deserves more, not less.

Second, 90 to 99% of the probability mass describes a future where humanity is fine but is dealing with industrial-scale deepfakes, autonomous weapons, labor displacement, surveillance, and concentration of cognitive power in three or four companies. That world deserves more policy attention than it currently gets. Skynet is more cinematic and the press likes cinema.

The interesting question is not "will AI kill us." It is: given a small but non-trivial tail risk, what specific governance and technical work matters most? The honest answers are unglamorous — mandatory pre-deployment evaluations of frontier models, licensing thresholds tied to training compute, interpretability research that lets us see what models actually compute, watermarking and provenance standards for synthetic media, enforceable international agreements on autonomous weapons. None of this trends on social media. All of it is how the problem actually gets handled, if it gets handled.

What this means for you

If you build software: the right question is not whether to use AI, but where you sit in the loop. Systems that keep a human authorizing any consequential action are durable. Systems that remove the human are brittle, both technically — the model is still wrong 5-15% of the time on hard tasks — and politically, since regulators will eventually catch up and you do not want to be on the wrong side when they do.

If you invest: watch capital expenditure on AI infrastructure, not headlines about AGI. NVIDIA's data-center revenue crossed $30 billion per quarter in 2024 and shows no sign of cooling. The story of the next decade is the build-out of a cognitive layer of the economy. The singularity is a distraction from the actual trade.

If you vote, read, or teach: the question of who controls frontier models matters more than the question of whether they will become conscious. Three or four labs — OpenAI, Anthropic, Google DeepMind, plus Meta and xAI in the chasing pack — will train the systems that mediate most of the world's information by 2030. The governance question — who they answer to, what they refuse, what they record about you — is where the genuinely consequential fights of this decade live.

You don't need to settle on a probability for extinction. You need to know that the people who have settled on one are not crazy at either pole, and that the boring question — what to do about deepfakes, weapons, jobs, and the concentration of cognitive power in five Californian companies — is the one that will define the next ten years whether or not the long tail ever arrives.

Ask Gradient

Want to go deeper? Use Gradient's Ask Gradient RAG product — retrieval-augmented generation, meaning the model answers from a curated database of papers and primary sources rather than its own memory — to chase the actual arguments. Try a question like "What did the Bletchley Declaration commit signatory countries to do about frontier AI risk?" or "What is the difference between alignment and corrigibility, and why does it matter?" Every answer cites its sources, so you can read Bostrom or Yudkowsky or the survey data yourself rather than my version of them. If you want the foundations first, start with our guide on what an LLM actually is and work outward.