Embeddings: The Coordinate System Quietly Running Modern AI

When Google knows you meant "trainers"

Type "pre-loved sneakers" into Google and the second result will say "used trainers". The two phrases share zero words. The retrieval system knows they share a meaning — because somewhere, an algorithm has quietly placed them at almost the same point in a 1,536-dimensional space.

This guide is about that space. Embeddings — the practice of representing meaning as a list of numbers — sit underneath nearly every useful AI feature shipped in the last five years. They do not generate poems or pass the bar exam. They are the plumbing on which search, recommendation, clustering and RAG run. By the end you should understand why a doctor's note and a patient's complaint can land on the same coordinate, and why that fact was worth a billion-dollar industry.

An embedding is a coordinate

Take the word "cat". An embedding model — a neural network trained to compress text into a fixed-length list of numbers — turns it into something like [0.21, −0.84, 0.07, …] with anywhere from 384 to 4,096 entries. That list is the embedding. It is not metadata about the word. It is the word, in the geometry the model has learned.

Why a list of numbers? Because numbers can be compared. Two embeddings can be subtracted, averaged, or measured for distance. The most famous demonstration came from Word2Vec — Tomáš Mikolov's 2013 paper at Google — which showed that the embedding for "king" minus "man" plus "woman" lands closer to "queen" than to any other word in the vocabulary. Meaning, it turned out, has algebra.

Word2Vec embedded individual words. Modern systems — OpenAI's text-embedding-3-large, Cohere's Embed v3, Google's Gemini embeddings — embed entire sentences, paragraphs, or documents. The principle is the same: every chunk of text becomes a single point in a high-dimensional space, and that point encodes what it means. Sentences with the same intent — "how do I cancel my subscription?" and "I want to stop paying for this" — land at almost the same coordinate, even though they share one common word.

Why this works at all

The reason embeddings encode meaning is unromantic. The model is trained on a simple task: given a piece of text, predict the surrounding text. A network that gets good at this has no choice but to develop an internal representation in which "physician" and "doctor" end up close — they appear in the same contexts. "Bank" by a river and "bank" with a vault drift apart, because the contexts diverge.

Linguists called this the distributional hypothesis decades before deep learning made it computable. J. R. Firth's 1957 line — "you shall know a word by the company it keeps" — is the entire theory of embeddings in eight words. The neural network is just a very expensive implementation.

What most people miss is that the coordinates are uninterpretable. Dimension 47 does not mean "royalty" or "feminine". The model learned a useful geometry, not a labelled taxonomy. We can measure distances in this space; we cannot read it. Treating an embedding vector as a list of features is the most common rookie mistake — it is a location, not a description.

What embeddings actually unlock

Once meaning is a coordinate, four things become trivial that used to be hard.

Semantic search. Traditional search ranks documents by keyword overlap — TF-IDF, BM25, the algorithms inside Elasticsearch. They are fast and dumb. Embedding-based search compares the query's coordinate to every document's coordinate and returns the nearest neighbours. "Pre-loved sneakers" retrieves a page about "used trainers" because the embeddings know those mean the same thing. Notion, Slack, and most modern customer-support tools now run a hybrid of both.

Recommendations. Spotify embeds songs based on listening patterns; tracks listened-to-by-the-same-people end up close in the space. Discover Weekly is, in effect, a nearest-neighbour query around your recent listens. YouTube does the same for videos, Pinterest for images, Amazon for products. The algorithm is the same algorithm — only the embedding model differs.

Clustering and deduplication. Embed a million customer-support tickets, run a clustering algorithm, and topics emerge without anyone defining them. The same trick is used to deduplicate news articles ("is this the eighth rewrite of the same Reuters wire?") and to organise vast archives of legal filings, GitHub issues, or scientific papers.

Retrieval-augmented generation, or RAG — the technique of feeding a large language model relevant documents at query time so it can answer with current or private information. RAG is currently the most common production use of LLMs inside enterprises, and it is built on embeddings end-to-end. The user's question is embedded, the nearest documents are pulled from a vector database, and the model is handed those documents as context. Hallucinations are not solved, but they are constrained — the model is now reasoning over real text instead of guessing from training data.

The original RAG paper is Lewis et al., Meta FAIR, 2020. Five years later, almost every internal chatbot at every Fortune 500 — every "chat with your docs" product — is a RAG pipeline. The acronym is fashionable; the architecture is plumbing.

Vector databases, and the industry that grew around them

Once you have ten million embeddings, you cannot just compare a query to all of them. Linear search across a 1,536-dimensional space at scale will burn a CPU. The problem is approximate nearest neighbour (ANN) search — finding the closest vectors fast enough to power a live application.

This is what vector databases solve. Pinecone (founded 2019, over $100M raised), Weaviate, Qdrant, Milvus, Chroma, and the open-source pgvector extension for PostgreSQL all implement variations of the same core algorithms — HNSW (Hierarchical Navigable Small World, Malkov & Yashunin, 2016) is the most common. They build an index in which finding the 10 nearest neighbours to a query is logarithmic in the number of vectors, not linear.

The category briefly looked like it would be the database story of the decade. Pinecone reached a roughly $750M valuation in 2023. Then a quiet correction set in. pgvector turned out to be good enough for most workloads under a hundred million vectors, which is most workloads. By 2024, MongoDB, Redis, Elasticsearch, and Snowflake had all shipped native vector search. The standalone vector database is no longer the obvious answer.

What most teams reaching for Pinecone actually need is a column in their existing Postgres. If your data fits on one machine, pgvector retrieves correctly and lets you join against the rest of your data in the same query. The pure-play vector databases will keep the ultra-large, high-throughput tier — the OpenAI-scale customers — and that tier is smaller than the 2023 hype suggested.

The frontier — one space for everything

Until recently, every modality had its own embedding space. Text models embedded text. Vision models embedded images. The two could not be compared. This changed in 2021 with OpenAI's CLIP — Contrastive Language-Image Pre-training — which trained a single model on 400 million image/caption pairs from the open internet, learning to place an image and its caption at nearly the same coordinate.

The implications were larger than they first looked. With one shared space you can search images using text ("find the photo where I'm holding a red mug"), retrieve captions using images, and feed both into a downstream system without translating between modalities. CLIP is the embedding layer underneath DALL-E, Stable Diffusion, and most modern image search.

Multimodal embeddings have since extended to audio (speech and music), video (Google's Gemini ingests it natively), and even biological data — AlphaFold's protein embeddings represent amino-acid sequences in a space where structural similarity is just distance. The pattern is general: any modality with enough paired data can be projected into a shared coordinate system with text.

The interesting question is what happens when meaning becomes universally addressable. A medical record, an X-ray, a transcribed consultation and a published paper can in principle all map to the same space — searchable by any of them, comparable across all. We are roughly halfway there. The bottleneck is no longer the embedding model. It is the data.

What this means for you

If you build software, embeddings are the cheapest AI feature you can ship. Calling OpenAI's text-embedding-3-small costs around $0.02 per million tokens — orders of magnitude cheaper than calling GPT-4. A semantic search bar over your own help docs is a weekend project, not a quarter-long one. If you are not using embeddings somewhere in your product, you are leaving the lowest-hanging fruit in modern software on the tree.

If you invest, watch where the value accrues. The embedding models themselves are commoditising — OpenAI, Cohere, Voyage, and open-source models like BGE and E5 are all within a few percentage points on the standard benchmark, MTEB. The infrastructure layer is consolidating into existing databases. The durable value will be in vertical applications — legal RAG, medical RAG, internal knowledge graphs — where the moat is proprietary data, not the model.

If you are simply trying to understand what is going on, the mental model is this: every AI system you interact with — search, recommendations, chatbots, image generation — has an embedding layer somewhere underneath, quietly turning the messy world of language and images into clean coordinates that machines can manipulate. The LLM gets the headlines. The embedding does the work.

Ask Gradient

For a deeper dive — how embedding dimensions are chosen, why some models use 384 and others 4,096, what Matryoshka embeddings unlock — try our Ask Gradient RAG search, trained on eighteen months of AI research and reporting. A good question to start with: "What is the actual trade-off between embedding dimension and retrieval accuracy?"