Drift Detection: Docs That Are Tested Like Code

The Count That Lied

A learning platform's README once stated a lesson count that was accurate when it was first written. Over the following two years, the platform grew — new tracks, new lessons, the whole catalog expanding — but the number in the README was a hand-written fact that required a human to remember to update it. Nobody did. The README eventually bragged a count that was more than three hundred lessons behind reality.

The count was discovered during a platform review. The fix someone considered first was finding the right number and updating the file. That is the wrong fix.

The right fix is to delete the hand-written number entirely and replace it with a test that asserts the real one in CI. A count-integrity test that scans the actual lesson files on disk, counts them, compares against the registry, and fails the build when they diverge. Not a warning. Not a badge. A hard failure that blocks merge until the registry and the disk are back in sync.

That is the fundamental insight of drift detection: a doc that states a countable fact should either compute it or be tested against it. If the number lives in a file that a human updates, it is a future lie.

Drift Classes and Their Detectors

Documentation drift is not one problem — it is five distinct problem classes, each requiring a different detector. Understanding the class tells you which tool to reach for.

Countable facts are the simplest class to detect and the most common source of quietly inaccurate documentation. Any number that could be derived by counting something — lesson counts, endpoint counts, environment variable counts, config key counts — is a countable fact. The detector is a count-integrity test: scan the source, count it, assert it matches the documented figure. This academy's own CI pipeline uses this pattern directly: it asserts the exact lesson count, validates every lessonOrder entry against files on disk, and cross-checks BuildChallenge IDs against the registry. These are count-integrity tests that BLOCK merge — not warnings, not advisory, not soft.

Links and paths drift when files are renamed, directories are moved, or content is deleted. The README key-files table, the runbook paths, the internal anchor links in your documentation — all of these reference filesystem paths or URLs that can stop resolving without any change to the doc itself. The detector is a link checker: a vitest test that reads the table, resolves each path against the repo root, calls fs.existsSync on each one, and fails with a list of missing paths. This class is inexpensive to test and catches a large fraction of silent drift from refactors.

Schemas and contracts drift when an API field is renamed, a TypeScript interface is updated, or a JSON schema evolves. The documentation may still compile and render correctly — the drift is semantic, not structural. The detector is a schema drift test: regenerate the TypeScript types from the current schema, diff against the committed types file, fail if they diverge. This makes a field rename in the schema and a corresponding doc update a forced pair rather than a convention.

Code examples in docs drift when the API they demonstrate changes. A code block showing client.messages.create({ model: "old-model-name" }) compiles and renders fine — the drift is only visible when someone runs it. The detector is an extracted-snippet compilation test: parse the fenced code blocks from markdown files, write them to temporary TypeScript files, run tsc --noEmit, fail on any error. This class requires more instrumentation than the others but protects against the drift that is most immediately visible to someone following the docs.

Command and flag references are the drift class most likely to cause an incident. A runbook step that says deploy --blue-green references a flag that the deploy tool may remove in a future version. The detector is an existence check: spawn each referenced command with --version or --help, assert exit 0. For runbook steps, run the safe verify-steps on a schedule — not just on PR. A tooling upgrade that removes a flag does not show up in a PR diff; it shows up during an incident when someone follows the runbook verbatim.

Writing Drift Tests Like Any Other Test

The tooling change that makes drift tests tractable is treating them as first-class vitest tests. They live in __tests__/, they run on every PR, they fail with readable error messages, and they are maintained alongside the code they protect.

The count-integrity flow diagram shows the simplest version: scan the directory, count the files, compare against the registry count, fail with a hard exit if they diverge. The GitHub Actions step that calls npx vitest run fails the workflow on any non-zero exit. The PR cannot merge.

Here is the pattern in full for a lesson count test:

// __tests__/content-integrity.test.ts
import { describe, it, expect } from 'vitest'
import * as fs from 'fs'
import * as path from 'path'

const CONTENT_DIR = path.resolve(__dirname, '../content/academy')

describe('content-integrity', () => {
  it('lesson file count matches registry entry count', () => {
    const files = fs.readdirSync(CONTENT_DIR)
    const realCount = files.filter((f) => f.endsWith('.mdx')).length

    // In a real repo, import TRACKS from 'lib/tracks' and count lessonOrder entries
    // Here we read a simple registry file for the exercise
    const registryCount = JSON.parse(
      fs.readFileSync(path.resolve(__dirname, '../lib/lesson-registry.json'), 'utf8')
    ).length

    expect(realCount).toBe(registryCount)
  })
})

The link drift pattern is equally direct:

describe('link-drift', () => {
  it('every path in the README key-files table exists on disk', () => {
    const readme = fs.readFileSync(path.resolve(__dirname, '../README.md'), 'utf8')
    // Extract paths from the key-files table (simplified extraction)
    const pathPattern = /\|\s+`([^`]+)`\s+\|/g
    const paths: string[] = []
    let match: RegExpExecArray | null
    while ((match = pathPattern.exec(readme)) !== null) {
      paths.push(match[1])
    }

    const missingPaths = paths.filter(
      (p) => !fs.existsSync(path.resolve(__dirname, '..', p))
    )
    expect(missingPaths).toEqual([])
  })
})

Scheduling and Scope

Not all drift tests run on the same cadence. Count integrity and link drift run on every PR — they are fast, stateless, and protect against the most common class of structural drift introduced by normal development.

Schema drift tests run on every PR as well, since a field rename in the same commit is the most common source of schema drift.

Command-reference tests are more expensive — spawning processes takes time — and some referenced commands can only be verified in an environment that has the tooling installed. These run on a daily schedule in a prepared environment, not on every PR. A failure pages the on-call engineer with a link to the failing test and the runbook step it covers.

The scope for each class of drift test should be narrow. A count-integrity test that asserts the lesson count is a one-line assertion — not a test that verifies lesson content, not a test that validates YAML frontmatter. One test, one invariant. When the test fails, the failure message tells you exactly what drifted without requiring investigation.

The BuildChallenge for this lesson implements both patterns from scratch: a count-integrity test and a link drift test, both as vitest tests, both structured to produce maximally useful failure messages.