RAG in Production
The retrieve → rerank → cache → generate path, built the way Semantic Memory Layer actually runs it: local MiniLM embeddings, ChromaDB, and a semantic cache that pays for itself.
RAG Is a Pipeline, Not a Prompt
The demo version of retrieval-augmented generation is one embedding call, one vector search, and a stuffed prompt. The production version is a pipeline with six distinct hops, each of which can fail independently and each of which has a cost. Getting RAG right in production is mostly about respecting those seams.
Semantic Memory Layer is a working example of every hop. It indexes 2,968 chunks across 43 repos, answers queries in under 200ms, and serves Knox's Mission Control dashboard and every Claude Code session. None of it touches a hosted embedding API.
Local Embeddings — Not a Compromise, a Default
The pipeline's second hop embeds the query. Knox's entire stack does this with all-MiniLM-L6-v2, a local model that produces 384-dimensional vectors on CPU in single-digit milliseconds. This is not a budget choice forced by frugality — it is the correct default.
Retrieve Broad, Rerank Narrow
The single most common production-RAG failure is injecting confidently-wrong context. Vector search ranks by cosine similarity over independently-computed embeddings, which is fast but fuzzy — it routinely surfaces chunks that are topically near but not actually relevant.
The fix is two-stage retrieval. Over-fetch with the cheap, fuzzy vector search (k=20), then rerank that shortlist with a cross-encoder that scores each (query, chunk) pair jointly. The cross-encoder is slow, which is exactly why it only runs on the 20 candidates, not the whole index. Keep the top 5, inject those, generate. Skip this hop and the model answers fluently from the wrong source.
Three Caches, Checked in Order
Every hop in the pipeline costs something. Caching is how you stop paying for the same answer twice — and there are three different kinds of "same," so there are three caches.
The response cache catches byte-identical repeats and is nearly free to serve, but in conversational traffic almost nothing is byte-identical. The embedding cache catches re-ingestion of unchanged document chunks — a chunk's vector never changes until the chunk does. The one that actually pays for RAG is the semantic cache: it embeds the incoming query and, if it is close enough to a previously-answered query, returns the stored answer before the expensive LLM call. Paraphrases become hits.
Wiring the Cache-First Generate Path
Production RAG checks the semantic cache before it spends a single LLM token. The control flow is: embed the query, check the semantic cache, and only on a miss run the full retrieve → rerank → generate path — then populate the cache on the way out.
What's Next
You can serve a model (the previous lesson) and ground it in retrieved knowledge with a cache that controls cost (this lesson). The next lesson makes the cost explicit: inference economics — quantization, prefix caching, spot instances, and scale-to-zero, ranked by dollar impact against the effort and quality risk of each.