The Re-Embed Lifecycle: Your Index Is Always Stale

The Index Is a Cache

The vector index is not the source of truth. The corpus — your documents, your lessons, your knowledge base — is the source of truth. The index is a derived artifact: a transformation of the corpus into vector space that enables similarity search. Like every derived artifact, it drifts from its source the moment the source changes.

Two incidents illustrate what this drift costs.

The first: an AI Q&A assistant on a personal site had one document in its corpus containing a real credential. The assistant reproduced it in answers. The credential was identified. The document was redacted. The fix was merged. Users continued receiving the credential in answers for days. The fix had changed the corpus — it had not changed the index. The old chunks, with the old content including the credential, still lived in the vector store. Retrieval continued serving them. The rule: redaction completes at re-embed, not at merge.

The second: a production tutor's lessons were rewritten to reflect updated content. Learners kept getting answers from the old versions. The ingest job's process health check returned healthy — 200 OK, process alive. The index had not been updated since before the rewrites. Process liveness said the system was healthy. Artifact freshness — the actual timestamps on chunks in the index — said the system was three weeks stale.

Both failures have the same root cause: the index was treated as a live system component when it is actually a cache. Every cache needs invalidation. The re-embed lifecycle is that invalidation.

Checksum-Based Incremental Re-Embedding

Re-embedding the entire corpus on every run is correct but expensive. For a corpus of 10,000 documents, that is 10,000 embedding API calls per run, regardless of whether 9,953 of those documents have not changed since yesterday.

The solution is a content hash per document stored at embed time. On each run, compare the current hash to the stored hash. Only re-embed when they differ.

// lib/checksum-reembed.ts
import crypto from "crypto"

function documentChecksum(content: string): string {
  return crypto.createHash("sha256").update(content).digest("hex")
}

async function detectChanges(
  docs: SourceDocument[],
  namespace: string
): Promise<{ changed: SourceDocument[]; unchanged: number }> {
  // Batch query stored checksums from the chunks table
  const docIds = docs.map((d) => d.doc_id)
  const result = await pool.query(
    `SELECT DISTINCT ON (doc_id) doc_id, checksum
     FROM chunks
     WHERE doc_id = ANY($1) AND namespace = $2
     ORDER BY doc_id, updated_at DESC`,
    [docIds, namespace]
  )

  const stored = new Map(result.rows.map((r) => [r.doc_id, r.checksum]))

  const changed: SourceDocument[] = []
  let unchanged = 0

  for (const doc of docs) {
    const newHash = documentChecksum(doc.content)
    if (stored.get(doc.doc_id) === newHash) {
      unchanged++
    } else {
      changed.push({ ...doc, _newChecksum: newHash })
    }
  }

  return { changed, unchanged }
}

For the 10,000-document corpus where 47 documents changed, the job makes roughly 47 embedding calls instead of 10,000 — a more-than-200x reduction for that run. At the per-token cost of text-embedding-3-small, that difference compounds: qualitatively, embedding costs scale linearly with the number of chunks processed. Incremental re-embed keeps that cost flat between corpus changes.

The Delete-Then-Insert Pattern

When a document changes, its chunks change. If the content was split into 8 chunks at the previous embed, and the rewrite produces content that splits into 6 chunks, two chunk slots are now orphaned. They contain stale content and will continue being retrieved until explicitly removed.

The pattern is: delete all existing chunks for the changed document before inserting the new ones — and wrap the whole swap in a transaction so a crash can never leave the document half-replaced.

// For each changed document, run the delete + re-insert as one transaction:
const client = await pool.connect()
try {
  await client.query("BEGIN")

  // 1. Delete stale chunks
  await client.query(
    "DELETE FROM chunks WHERE doc_id = $1 AND namespace = $2",
    [doc.doc_id, namespace]
  )

  // 2. Re-chunk the document
  const textChunks = splitChunks(doc.content)

  // 3. Embed and insert each new chunk
  for (let i = 0; i < textChunks.length; i++) {
    const embedding = await embedText(textChunks[i])
    const chunkId = `${doc.doc_id}_chunk_${i}`

    await client.query(
      `INSERT INTO chunks
         (chunk_id, doc_id, source_id, namespace, content, embedding, checksum, updated_at)
       VALUES ($1, $2, $3, $4, $5, $6, $7, NOW())
       ON CONFLICT (chunk_id) DO UPDATE SET
         content = EXCLUDED.content,
         embedding = EXCLUDED.embedding,
         checksum = EXCLUDED.checksum,
         updated_at = NOW()`,
      [chunkId, doc.doc_id, doc.source_id, namespace, textChunks[i],
       JSON.stringify(embedding), documentChecksum(doc.content)]
    )
  }

  await client.query("COMMIT")
} catch (err) {
  await client.query("ROLLBACK")
  throw err
} finally {
  client.release()
}

The transaction boundary is what makes a crashed run safe to restart. If the job dies mid-document, the ROLLBACK discards the partial delete-and-insert, so the document's old chunks — and the old checksum they carry — remain intact. On restart, detectChanges still sees the old checksum and re-processes the document from scratch. Without the transaction, a crash after the DELETE but partway through the inserts would leave some chunks already carrying the new checksum; the restart's change detection could read that new checksum, classify the document as unchanged, and skip it — leaving it permanently missing chunks. The ON CONFLICT DO UPDATE handles chunk-ID collisions on re-insert; it is the transaction, not the conflict clause, that makes re-running safe.

Freshness as a Health Metric

Process liveness is not freshness. Liveness tells you the job is running. Freshness tells you the job is doing work that matters.

The freshness metric for a RAG index is MAX(updated_at) from the chunks table, compared to the last_modified timestamps of the source documents. When the gap between those two values exceeds your SLO, you have a stale index — regardless of what your process health check says.

// Freshness check — query the artifact, not the process
async function indexFreshnessSLOCheck(
  namespace: string,
  maxStalenessSecs: number
): Promise<{ fresh: boolean; staleSecs: number; lastUpdatedAt: Date }> {
  const result = await pool.query(
    "SELECT MAX(updated_at) AS last_updated FROM chunks WHERE namespace = $1",
    [namespace]
  )

  const lastUpdatedAt: Date = result.rows[0]?.last_updated ?? new Date(0)
  const staleSecs = (Date.now() - lastUpdatedAt.getTime()) / 1000

  return {
    fresh: staleSecs <= maxStalenessSecs,
    staleSecs,
    lastUpdatedAt,
  }
}

// This is the check your monitoring calls — not /health returning 200
// If staleSecs > maxStalenessSecs, page: the index is serving outdated content

This academy runs this check on every lesson corpus deploy. If MAX(updated_at) on the lessons namespace does not advance within the expected window after a lesson merge, an alert fires. The alert is not "ingest service is down" — it is "index is stale, learners are getting old answers."

When Full Re-Index Is Required

Checksum-based incremental re-embed handles one type of change: the same chunking strategy, the same embedding model, different content. Two changes require a full re-index:

Chunking strategy change. If you switch from fixed-size chunks to structure-aware chunks that split on heading boundaries, the chunk boundaries for every document change. Old chunks in the index were produced by the old strategy. New queries produce embeddings based on the new chunking vocabulary. The index must be rebuilt entirely.

Embedding model change. Changing from text-embedding-3-small to any other model changes the coordinate space. A chunk embedded under the old model and a query embedded under the new model are measuring distance in incompatible spaces. The model lock-in lesson from 579 makes this concrete: changing embedding models means re-embedding everything.

Full re-index is an all-or-nothing operation: delete the entire namespace, re-chunk everything, re-embed everything, rebuild the HNSW index. The cost is the full corpus at the per-token embedding rate. Budget for it before making chunking or model changes.

The CI Hook

This academy's lesson delivery pipeline wires re-embedding to the CI/CD process directly: when a lesson MDX merge lands on main, a post-merge hook triggers the ingest job for the lessons namespace. The job runs checksum detection, finds the changed lessons, re-embeds them, and verifies MAX(updated_at) advances. If it does not advance within the expected window, the deployment is flagged.

The result: every lesson update is reflected in the AI tutor's answers within the same deployment window. The freshness SLO is enforced by the pipeline, not by hoping someone remembers to trigger re-embed manually.

// .github/workflows/ingest.yml equivalent logic
// Triggered on push to main when content/academy/** changes

async function postMergeIngest(changedFiles: string[]) {
  const lessonFiles = changedFiles.filter(
    (f) => f.startsWith("content/academy/") && f.endsWith(".mdx")
  )

  if (lessonFiles.length === 0) return

  const docs = await loadLessonDocs(lessonFiles)
  const result = await incrementalReembed({ docs, namespace: "lessons" })

  const freshness = await indexFreshnessSLOCheck("lessons", 3600) // 1h SLO
  if (!freshness.fresh) {
    throw new Error(
      "Freshness SLO breached after ingest: " + freshness.staleSecs + "s stale"
    )
  }

  console.log("Ingest complete:", result.reembedded, "docs re-embedded,",
    result.unchanged, "unchanged")
}

What's Next

The next lesson builds the evaluation framework that validates all the retrieval, generation, and freshness work you have done across the last three lessons. Recall@K, mean reciprocal rank, faithfulness scoring, and the golden set that must hold on every corpus change — the measurement discipline that turns your RAG system from something you believe is working into something you can prove is working.