Chunking: The Decision That Haunts Every Query

The Decision You Can't Take Back

Every chunk you embed is a bet. You are betting that the text you are splitting at — whether a character boundary, a sentence boundary, or a heading — is the right place to divide the document for your specific query distribution. You are betting that the chunk size you chose will be precise enough to retrieve well and rich enough to generate from.

What makes chunking consequential is not the decision itself — it is the cost of changing it. Chunk size is the one parameter in a RAG pipeline that requires a full re-embed of the entire corpus to change. Not a partial update. Not an in-place resize. Every document gets re-split and every chunk gets re-embedded from scratch.

The practical consequence: teams that choose chunk size by guessing, deploy to production, and then discover retrieval quality problems face a non-trivial re-indexing bill to fix it. The right time to reason carefully about chunk size is before the first production embed, using a small golden set to measure retrieval quality at different sizes.

The Tradeoff Curve

Chunk size sits on a tradeoff curve between two goods that move in opposite directions. Smaller chunks give you retrieval precision: the matched text is tightly relevant to the query, with minimal off-topic content diluting the similarity score. But small chunks starve generation: the model receives a correct fragment without the surrounding argument, background, or definitions needed to produce a complete answer.

Larger chunks give the model rich context: the full section, with its introductory framing, supporting examples, and concluding summary. But they dilute similarity: a 1,000-token chunk about "chunking strategy, including a discussion of HNSW indexes, embedding models, and cost considerations" has a similarity score that is the average of all those topics — worse than a 300-token chunk that is specifically about chunking strategy.

Neither extreme is right for most corpora. The production starting point for structured prose documents is 400–600 tokens with 10–15% overlap. The overlap catches sentences that span a chunk boundary — a definition that starts at the end of one chunk and continues into the next. Beyond 20% overlap, you are paying re-embed cost for redundant coverage without measurable retrieval improvement.

Fixed-Size vs Structure-Aware

Fixed-size chunking with overlap is the baseline. It is simple, uniform, and has no structural awareness. Every document gets split the same way regardless of whether it is a continuous narrative, a heading-organized tutorial, or a code-heavy reference. As a starting point for exploration, it is reasonable. As a production strategy for structured content, it is a mistake.

Structure-aware splitting recognizes that document organization is semantic information. An H2 heading signals a topic change. A code fence signals a technical unit that must be understood whole. A table is a structured comparison that loses its meaning when split mid-row.

The production rules for Markdown, which powers the academy's lesson corpus:

• A new chunk starts at every H2 heading. The heading itself anchors the chunk's metadata with a URL fragment (#corpus-design).
• Code blocks are never split. If a code block would push a chunk over the target size, the chunk is allowed to exceed the size limit to keep the code intact.
• Tables are kept as single units.
• Prose paragraphs within a section are split at sentence boundaries if the section exceeds the size target.

Chunk Metadata Is Not Optional

Every chunk that enters the index must carry metadata beyond its text and embedding vector. The metadata is what turns a retrieved chunk into a usable citation and what enables filtered retrieval.

The minimum metadata payload per chunk:

• source_id: the document this chunk came from, used to group chunks by document and to filter retrieval by source
• heading_anchor: the URL fragment pointing to the section in the original document, used by the citation UI to deep-link
• position: the ordinal position of this chunk within the document, used to fetch adjacent chunks for context expansion
• checksum: a hash of the chunk text, used to detect when a chunk has changed and needs re-embedding
• updated_at: the timestamp of the last re-embed, used for freshness monitoring

Missing metadata is a debt that compounds. A system built without checksum has no way to implement incremental re-embedding — every change requires a full corpus re-index. A system built without heading_anchor cannot produce deep-linked citations. A system built without source_id cannot enforce per-source access control.

Re-Chunking Is Re-Embedding

A consequence that surprises teams encountering it for the first time: any change to the chunking strategy requires discarding the entire current index and rebuilding from scratch.

This is because the embedding of a chunk and the chunk's text are inseparable. The vector in the index represents the semantic content of a specific text string. If that string changes — because the chunk boundaries changed — the existing vector no longer represents the current text. You cannot "update" a chunk boundary in the index; you must delete the old chunks and insert the new ones.

The cost of this operation is proportional to the size of your corpus and the price of your embedding model. For the academy's tutor, re-embedding all lesson content is a meaningful but manageable cost that runs in minutes. For a corpus of millions of enterprise documents, a full re-embed can be a significant engineering event with associated cost and downtime considerations.

This is why the re-embed lifecycle lesson covers the incremental re-embed lifecycle: for content updates (a changed document), you only need to re-embed the affected document's chunks. But for structural changes — chunk size, chunking algorithm, or embedding model — there is no incremental path. Plan for it, budget for it, and test the new chunking strategy thoroughly on a golden set before committing to a full production re-index.

The Academy Chunker

The academy's lesson chunker implements structure-aware splitting on MDX files. Each lesson becomes a set of chunks bounded by H2 headings, with code blocks preserved as atomic units. The metadata includes the lesson number (source_id), the heading anchor, and a SHA-256 checksum of the chunk text.

When a lesson is merged to the main branch, CI triggers the ingest job. The job checksums each new chunk against the stored checksum in the index. If the checksum matches, the chunk is skipped. If the checksum differs, the chunk is deleted from the index and re-embedded from the updated text. If the chunk is new, it is embedded and inserted. This incremental approach means a small lesson edit re-embeds a handful of chunks rather than the entire corpus.

The chunking decisions made for MDX — heading-boundary splits, intact code blocks, 15% overlap on prose — were validated against a golden set of 30 questions before being deployed to production. That validation process is the subject of the RAG-evaluation lesson.