Embeddings & the Index: The Semantic Layer
Embeddings are semantic coordinates — choose your model carefully because changing it means re-embedding everything, and choose pgvector by default because boring infrastructure is production infrastructure.
What an Embedding Is
An embedding is a point in a high-dimensional space. When you embed a chunk of text, you are asking an embedding model to assign that chunk a set of coordinates — one number per dimension — such that chunks with similar meaning land near each other and chunks with different meaning land far apart.
With 1536 dimensions (the output of text-embedding-3-small), "near" and "far" are defined by cosine distance: the angle between two vectors. Two chunks about the same topic have vectors that point in roughly the same direction, so their cosine distance is small. A chunk about RAG chunking strategy and a chunk about stock prices have vectors pointing in very different directions, so their cosine distance is large.
You cannot visualize 1536 dimensions. The conceptual diagram below projects semantic similarity into two dimensions for illustration, but the actual space has no axes you can name — each dimension captures some learned aspect of meaning that does not correspond to a human-legible concept.
Choosing the Embedding Model
The model choice for embeddings is a long-term commitment. The academy's tutor uses text-embedding-3-small from the OpenAI embeddings API. This model produces 1536-dimensional vectors and offers a strong quality-to-cost balance for text retrieval over structured prose.
The choice matters for one reason above all others: you cannot mix vectors from different models in the same index. The cosine distance between a vector from model A and a vector from model B is geometrically meaningless — the two models learned different coordinate systems. If you decide to switch models, you must re-embed every chunk in your corpus, replace every vector in the index, and validate retrieval quality on a golden set before trusting the new index in production.
This is why the model name in your embedding pipeline is not a configuration knob to tune casually. It is an architectural commitment. Document it in your source inventory alongside the dimension count (1536) and the date you first used it. When a better model is released, treat the migration as a planned re-indexing event, not a hot-swap.
The cost framing for embeddings stays qualitative: embedding cost is per-token and cheap per individual call, but applied to millions of chunks across a large corpus it becomes a meaningful budget line. Use qualitative reasoning — "how many chunks, how often re-indexed, what token count per chunk" — rather than inventing specific dollar figures.
pgvector: The Boring-and-Right Default
The vector store decision has a simple heuristic: if your app already runs Postgres, use pgvector. If it does not and your scale does not require a dedicated store, consider adding Postgres.
pgvector is a Postgres extension that adds a vector(N) column type and several distance operators, including <=> for cosine distance. You declare the column when creating the table:
CREATE TABLE chunks (
chunk_id TEXT PRIMARY KEY,
source_id TEXT NOT NULL,
content TEXT NOT NULL,
embedding vector(1536),
checksum TEXT NOT NULL,
updated_at TIMESTAMPTZ DEFAULT now()
);
You create the HNSW index:
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
And you query by cosine similarity:
SELECT chunk_id, source_id, content, embedding <=> $1 AS distance
FROM chunks
WHERE source_id = $2
ORDER BY embedding <=> $1
LIMIT 10;
That is the complete retrieval primitive. The <=> operator computes cosine distance between the query vector (passed as $1) and each stored vector. The ORDER BY returns the closest chunks first. The WHERE source_id = $2 restricts to a specific namespace — the mechanism for access control and multi-tenant isolation.
The HNSW Index
An HNSW (Hierarchical Navigable Small World) index makes vector similarity search sub-linear. Without an index, a cosine similarity query does a full table scan — computing the distance between your query vector and every stored vector. With an HNSW index, the search navigates a graph structure to find near neighbors without comparing against every point.
The tradeoff: HNSW builds a complex graph that requires more memory than a flat index. For production use at standard scales, this is the right tradeoff — query speed and recall quality justify the memory cost.
For very large corpora (tens of millions of chunks) or memory-constrained environments, IVFFlat offers a lower-memory alternative with slightly lower recall. For development and testing with small datasets, no index (exact nearest neighbor) gives perfect recall with acceptable performance.
The rule of thumb: start with HNSW in development, use HNSW in production. Only move to IVFFlat or a dedicated store when you have measured a specific performance constraint that HNSW cannot meet.
Namespaces via Metadata Filters
A single pgvector table can serve multiple namespaces — free-tier and paid-tier content, different tenants in a multi-tenant system, content organized by topic domain. Namespacing is implemented as a WHERE clause filter on a metadata column:
-- Free-tier query: only retrieve chunks tagged for free access
SELECT chunk_id, content, embedding <=> $1 AS distance
FROM chunks
WHERE tier = 'free'
ORDER BY embedding <=> $1
LIMIT 10;
-- Paid-tier query: access all chunks
SELECT chunk_id, content, embedding <=> $1 AS distance
FROM chunks
ORDER BY embedding <=> $1
LIMIT 10;
The namespace is a metadata column on the chunk, set at ingest time and enforced at query time. This is the pattern the corpus-and-leak-gates lesson established when discussing access control: what goes into the corpus and how it is tagged determines what can be enforced at retrieval. A free-tier user's query physically cannot surface paid-tier chunks if the WHERE clause excludes them.
Hybrid Search for Exact Identifiers
Vector similarity is remarkable at semantic matching — finding chunks about "credential rotation" even when a query says "how to change API keys" — but it fails systematically on exact identifiers. Error codes like ERR_4291, version strings like v1.4.2, and function names like getUserById are not semantically similar to anything. Their embedding vectors sit in isolation in the space, not clustered with related concepts.
The fix is hybrid search: run vector similarity for semantic matching and keyword search for exact identifier lookup, then merge and deduplicate the results before sending them to the retrieval ranking step. In Postgres, pg_trgm trigram indexes or full-text search (tsvector) provide the keyword search half.
For the academy's tutor, hybrid search would matter if users asked for a specific lesson by its exact title rather than by topic. A query like "the chunking lesson" should retrieve the chunking lesson directly, not rely on semantic similarity to the word "578." This is a known gap in the current implementation, and the exact hybrid search pattern is the straightforward fix.
The BuildChallenge for This Lesson
The challenge asks you to implement the three primitives this lesson is built on: an embedText function that calls the OpenAI embeddings API, an upsertChunk function that inserts or updates a chunk in the pgvector table, and a queryChunks function that queries by cosine similarity with an optional namespace filter. These three functions form the complete read-write interface of the vector store — every upstream and downstream lesson operates through them.