Capstone: A RAG Service With Receipts

What This Track Built

Ten lessons ago, you encountered a framing: a RAG system that cannot show where an answer came from, prove its index is fresh, and refuse when retrieval is thin, is a liability — not a feature. Every lesson in this track was building toward a system that can make all three of those claims.

This is the assembly lesson. Not an introduction to new concepts, but a precise account of how each component you built connects to the next, and what fails at each seam if you get the connection wrong.

The architecture diagram maps every component to the lesson that built it. Read it as a dependency chain: the ingest gate's decisions constrain what reaches the index; the index's quality constrains what retrieval can surface; retrieval's accuracy constrains what generation can honestly claim; generation's structured output contract constrains what the UI can display. Failure at any stage propagates to every stage downstream.

The Seven Components, Assembled

Ingest gate (the corpus-and-leak-gates lesson). Every document passes through the gate before embedding. The gate enforces source allow-listing — only documents from approved sources enter the corpus. It scans for PII, credentials, and internal names. The rule from the credential-leak story in the corpus-and-leak-gates lesson: redaction completes at re-embed, not at source fix. Fixing the source document does not fix the index. The gate must run at every ingest path, not just the primary one. A fork, a copy, a secondary import pipeline — each is a re-leak path if it bypasses the gate.

Structure-aware chunker (the chunking lesson). Chunks are split on structural boundaries — headings, code fences, paragraph breaks — not on fixed token counts. Each chunk carries its source id, heading anchor, and position in the document. These metadata fields are the connection between the vector index and the citation: when a chunk is retrieved, its heading anchor is what the citation points to. A chunk without provenance metadata is evidence you can retrieve but not cite.

Embed and upsert with checksums (the embeddings and re-embed lessons). Each chunk gets a checksum before embedding. The checksum is the mechanism for incremental re-embedding: only chunks whose content has changed since the last index run are re-embedded. This makes corpus updates cheap and fast instead of requiring full re-indexing. More importantly, it makes the index a faithful cache of the current corpus state — which is the only condition under which "my index is healthy" is a meaningful claim. The pgvector schema:

CREATE TABLE chunks (
  chunk_id TEXT PRIMARY KEY,
  doc_id   TEXT NOT NULL,
  content  TEXT NOT NULL,
  heading  TEXT,
  checksum TEXT NOT NULL,
  embedding vector(1536),
  tenant_id TEXT,
  tier      TEXT DEFAULT 'free',
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

The HNSW index makes approximate nearest-neighbor search fast at scale. The metadata columns — tenant_id, tier, checksum, updated_at — make the table queryable for freshness checks and ACL enforcement.

Threshold and rerank retrieval (the retrieval-quality and RAG-operations lessons). The retrieval layer runs in two passes. The first pass queries pgvector with a cosine distance threshold: embedding <=> $1 < (1 - threshold). Results below the threshold are treated as no-result. The second pass takes the top candidates — typically 20 — and reranks them using claude-haiku-4-5 to identify the most contextually relevant subset. The reranker is the layer that converts high-recall into high-precision retrieval.

The threshold is also the refusal trigger. If no results pass the threshold, the pipeline does not proceed to generation — it returns the refusal response. This is not an error state. It is the correct behavior: "I don't have that in my sources" is better than a confident hallucination. The threshold value is calibrated against your golden set — observe the score distribution on out-of-corpus questions and set the threshold above the noise floor.

Cited structured generation with refusal (the citations lesson). Generation receives the reranked chunks with their chunk ids. The structured output schema:

const schema = {
  type: "object",
  properties: {
    answer: { type: "string" },
    citations: { type: "array", items: { type: "string" } },
    confidence: { type: "string", enum: ["high", "medium", "low"] },
  },
  required: ["answer", "citations", "confidence"],
  additionalProperties: false,
}

The capstone flattens citations to a plain array of chunk ids to keep the assembled pipeline compact; production keeps the richer object form from the citations lesson — { source_id, chunk_id, quote } — so each receipt carries the exact quote it supports. Either way the citations array is a contract: every item must trace to a chunk id that was retrieved, and the claim it supports must appear in that chunk's content. Empty citations on a non-refused answer are a tripwire — something in the citation parsing broke, or the model is generating without citing. Alert on it; do not silently pass it through.

The Maturity Model

The maturity model gives you a precise vocabulary for where any RAG system sits and what it would take to move it up a level. Most production RAG systems are at L1 or L2. The gap from L2 to L3 is the eval gate and the security hardening — the infrastructure that converts the system from something you hope works into something that alerts you when it stops.

L0 is a prototype. Top-k retrieval with no threshold, no citations, no freshness mechanism. Correct approximately half the time by accident. The failure modes are invisible because nothing measures them.

L1 adds the threshold and structured output. The system can refuse when retrieval is thin. Citations make the answer auditable in principle. Re-embedding is manual and infrequent, which means staleness is the constant risk.

L2 is production-ready for single-tenant, single-tier systems. HNSW index, reranker, defensive citation parsing, incremental re-embedding, observability. The system knows when it retrieves well and when it refuses correctly. What it lacks is systematic regression detection.

L3 adds the eval gate, injection defense, and access control. The eval gate transforms corpus updates from hope-it-works changes into gated changes. Injection defense makes the system safe for corpora that might contain adversarial content. Access control makes multi-tenant deployment possible. An L3 system can be trusted with customer data.

L4 closes the compounding loop. Adversarial and out-of-corpus questions in the golden set. Faithfulness judging per answer in the eval pipeline. Prompt caching on stable prefixes. CI wired so a content merge triggers ingest and the eval gate before the change reaches production. The academy's own AI tutor operates at this level — lesson content merges trigger the full pipeline, and the eval gate verifies that learners getting answers about updated lessons are getting accurate answers.

What Breaks at the Seams

The most common failure points are not within components — they are between them.

Between ingest gate and chunker. A document passes the gate clean, then gets chunked in a way that splits a PII pattern across chunk boundaries. The gate scanned the whole document; the chunker produces fragments. Scan again at the chunk level before embedding, or ensure your chunker preserves natural boundaries that keep PII patterns intact.

Between chunker and index. A chunk's heading anchor is set at chunking time and embedded as metadata. If the document structure changes and the chunk is re-chunked without updating the anchor, citations point to the old heading. Checksums detect content changes; they do not detect structural changes. Include the heading in the checksum.

Between retrieval and generation. The reranker returns chunk ids. The generation prompt assembles chunk content. If the assembly is wrong — a slice bug, a missing separator, a chunk that gets truncated — the generation model reasons from corrupted context. Test the exact string that goes into the generation prompt against the raw chunks; the wire format is not always what you think it is. The dead source-chips bug from the citations lesson: a trailing ] in a slice turned JSON.parse into a silent failure for weeks.

Between generation and UI. Citations in the structured output are chunk ids. The UI maps them to source links. If the mapping is wrong — if the chunk id format changed, if the document metadata that the UI reads was not updated — the citations render as broken links. The receipt exists; the UI cannot show it. Test the full rendering path, not just the structured output shape.

Building It Yourself

The capstone challenge assembles the pipeline core: gated ingest through cited structured generation, with the incremental re-embed mechanism and the threshold-and-rerank retrieval wired in. This is the thread that runs through every lesson in the track.

Work through it in stage order: get the ingest gate running and confirm it rejects bad sources and catches the PII patterns. Then add the chunker and verify that a markdown document with headings produces correctly-attributed chunks. Then the embed-and-upsert step with checksum comparison. Then retrieval with threshold and reranking. Then generation with the structured schema and citation enforcement.

At each stage, verify the output before proceeding to the next. The seam between stages is where bugs hide. A chunker that produces correct output in isolation can still hand malformed chunk ids to the embed step. An embed step that upserts correctly can still return wrong metadata to the retrieval layer. Test the handoffs, not just the components.

The system you build in the challenge is not a tutorial exercise. It is a production skeleton. The mock functions — embedText, rawVectorSearch, runGeneration — are the interfaces you replace with real implementations. The real pgvector embed call, the real OpenAI embeddings API call, the real claude-sonnet-4-6 generation call. The structure holds; the implementations swap in.

This is the architecture that the academy's own AI tutor runs on. Every lesson you have read in this track was answered, if you asked a question about it, by a system built on this skeleton. The receipts you have been reading about are the receipts the tutor produces. The golden set it runs on includes questions about these exact lessons. The pipeline is real. Now you can build one yourself.