Corpus Design & Leak Gates: What Goes In Decides What Comes Out
The corpus is not a directory of documents — it is a curated, gated, access-controlled knowledge base where every source has an owner, a trust level, and a freshness cadence.
The Corpus Is the Foundation
Every answer a RAG system produces is bounded by what is in the corpus. A model cannot generate a faithful answer from a chunk that does not exist. It cannot attribute a claim to a source that was never ingested. It cannot refuse a question that falls outside the knowledge base if it has no signal that the question is outside the knowledge base.
This is the uncomfortable reality of corpus design: quality control at ingest time is not an optional refinement — it is the primary leverage point for answer quality. A carefully designed corpus with good chunking and mediocre retrieval will outperform a poorly designed corpus with excellent retrieval.
The source inventory is the starting point. Before embedding a single document, every source in the corpus needs to be characterized along four axes: trust level (is the content authoritative and vetted?), freshness cadence (how often does it change, and what triggers re-embedding?), license (are you allowed to index and serve this content?), and owner (who is responsible when the source has a problem?).
The Gate: Non-Negotiable
The ingest gate is the most important piece of infrastructure in a RAG pipeline. It runs before embedding. It runs on every document. It runs on every ingest path — not just the primary one.
The gate enforces three categories of checks:
Credential and secret scanning. API keys, tokens, connection strings, and passwords have no legitimate place in an AI knowledge base. Pattern matching against common credential formats (sk-, ghp_, password=) catches most cases. The gate treats any match as a hard block — the document does not enter the pipeline until the credential is removed and the gate re-run.
PII detection. Email addresses, phone numbers, and other personal identifiers that entered a document inadvertently must be caught before they become retrievable facts. A regex scanner covers the common patterns; more sensitive corpora may warrant an NER model pass.
Internal term denylist. Internal project codenames, machine hostnames, and infrastructure identifiers have no place in a public corpus. A configurable denylist checked against every document prevents inadvertent internal-to-external leakage.
The story that grounds this lesson: an AI Q&A assistant on a developer's personal site had exactly one document in its corpus containing a real API credential. The assistant happily reproduced it in answers to relevant questions. When the developer found the issue, they fixed the source document, committed, and merged. Two days later, users were still receiving the credential.
The fix came only when the developer realized that the index — a cache of the corpus — had not been updated. The old chunk, with the credential embedded in its 1536-dimensional vector, was still in the pgvector table. The fix required deleting the old chunks for that document from the index and running the ingest job from the clean source.
The Fork Problem
A fork of your content repository is a separate ingest path. A fork taken before a redaction was merged still contains the credential. If a colleague or downstream consumer is running their own RAG pipeline against a fork, their gate needs to run independently — and their index needs to reflect any redactions you made to the canonical source.
This is not a hypothetical edge case. In any organization where multiple teams build on shared content, forks are common. The canonical repo having a clean gate does not mean the fork does. Gate every ingest path you control, and document which paths you cannot directly control so the owners can be notified when redactions occur.
Document Hygiene
Beyond the security gate, every document benefits from a hygiene pass before chunking:
Strip boilerplate. Navigation HTML, cookie consent banners, footer text, and sidebar content add tokens that dilute the semantic signal in each chunk. A document with 200 tokens of substantive content and 800 tokens of navigation is a poor embedding target.
Normalize formatting. Multiple consecutive blank lines, inconsistent heading levels, and inline HTML mixed with Markdown all create chunking artifacts. A normalization step before chunking reduces these edge cases.
Preserve code blocks. Code blocks must survive the hygiene pass intact. They contain precise technical information that is often the primary reason a user's query matches a particular document. Any transformation that distorts code — removing special characters, normalizing whitespace inside fences — degrades the corpus.
Public and Private Corpus Separation
When your RAG system serves users with different access levels, the separation lives in the metadata, not in separate indexes. A single pgvector table can serve both free and paid users if every chunk carries a tier column. At retrieval time, the WHERE clause adds AND tier = $user_tier, and the query only considers chunks the user is authorized to see.
This matters for the generation step too. If paid-tier chunks appear in the model's context window during a free-tier user's query — even if they are filtered from the citations — the information may still influence the answer. Access control at retrieval is not just a citation-visibility concern; it is a content-isolation concern. The model must never see a chunk the user is not authorized to retrieve.
The RAG-operations lesson covers the full access-control implementation. This lesson establishes the principle: what goes into the corpus and how it is tagged determines what can be enforced at retrieval. Retroactive access control against an untagged corpus requires a full re-index.
Building the Source Inventory
The source inventory is a living document — a table maintained alongside the corpus configuration that records every source, its trust level, freshness cadence, license status, and owner. It is not a one-time audit; it is updated when sources are added, removed, or when a source's ownership or license status changes.
For the academy's tutor, the source inventory has one primary entry: the lesson MDX files, trusted (first-party), re-embedded on every merge to the main branch, proprietary license, content team as owner. Additional sources — SDK documentation, public reference materials — would each get their own row with their own cadence and owner.
Start the source inventory before writing a single line of ingest code. It is the product decision that determines which documents deserve engineering effort to ingest correctly, which require legal review before indexing, and which should be excluded entirely.