Runbooks That Stay True

The Runbook That Aged Out

A deploy runbook's step 3 read: deploy-tool --legacy-mode ./dist. An on-call engineer at 3am followed it verbatim. The error message was immediate: unknown flag: --legacy-mode. The deploy tool had been upgraded to v3.0 sixty-two days earlier, and that version had removed the flag. The runbook had been through a quarterly doc review during those sixty-two days. The reviewer had confirmed the file existed and updated verified_at. Nobody had run the steps.

Twenty minutes passed while the engineer read the deploy-tool changelog, identified the removed flag, reconstructed the correct invocation, and verified the deploy succeeded. The runbook had predated the tooling change by two months.

The lesson-543 prerequisite track taught what a runbook's anatomy should look like and why verification sections must prove the artifact changed, not just that a process is running. This track builds the automation that makes stale runbooks detectable before the 3am call. The machinery here is drift detection applied to a specific document type whose failure mode has a human cost.

Runbooks Are Executable Claims

Every numbered step in a runbook is an assertion about the current state of the system: that a command exists, that a flag is valid, that a path is reachable. These assertions were true when the runbook was written. They may not be true now.

The difference between a runbook and a static document is that runbook steps are intended to be executed. That makes their claims falsifiable in a way that a narrative architecture doc's claims are not. You can programmatically verify whether --legacy-mode exists in the deploy tool's help output. You can check whether ./dist is a valid path in the repo. You can confirm that npm ci is a valid command in the project's tooling context.

This is the insight: because runbook steps reference concrete system artifacts, those artifacts can be checked by a CI job on a schedule. The check is not running the deploy — it is proving that the deploy would not fail on a reference that no longer exists.

The verification manifest makes this concrete. A checked-in JSON file records which commands are valid, which flags each command accepts, and which paths are expected to exist. When the deploy tool drops a flag in v3.0, the manifest is updated in the same PR. The drift-check job then catches every runbook that referenced the removed flag, before anyone follows one at 3am.

What a Runbook Drift Test Checks

Four categories of references in runbook steps can be verified without executing the deploy:

Commands. deploy-tool, npm, psql, gh — these are executables. A drift test checks whether each command name appears in the manifest's valid-commands list. When a tool is renamed, sunset, or replaced, the manifest is updated and the job catches every runbook that still references the old name.

Flags. --legacy-mode, --dry-run, --force — these are command-specific. A drift test checks whether each flag appears in the manifest's per-command flag list. Flag removals are a common source of runbook drift because they happen silently: the tool's help output changes, but no migration document tells you which runbooks referenced the old flag.

Paths. ./dist, /etc/app/config.yaml, data/trades.db — these are file system claims. A drift test checks whether each path appears in the manifest's known-paths list. Paths change when directory structures are reorganized, when build outputs move, or when configuration files are renamed.

Expected outputs. The verify section of a runbook often says: "Expected: Already up to date." These are harder to check automatically and are addressed by the verification section contract from the prerequisite track. The drift-checker focuses on the three categories above because they are machine-verifiable without executing the deploy.

The Manifest Is the Contract

The verification manifest is not a generated file — it is a checked-in declaration of what the system currently supports. When a tool changes, the manifest must be updated in the same PR. The manifest update is the point-of-change signal that triggers the drift check.

// docs/runbooks/manifest.json (checked in, updated when tooling changes)
{
  "commands": ["deploy-tool", "npm", "psql", "gh"],
  "flags": {
    "deploy-tool": ["--dry-run", "--env", "--rollback"],
    "npm": ["ci", "run", "install"],
    "gh": ["pr", "repo", "release"]
  },
  "paths": ["./dist", "./scripts/deploy.sh", "data/"]
}

The manifest plays the same role as a schema in contract testing: it is the agreed-upon surface that runbooks are tested against. This academy's CI asserts the exact lesson count against the files on disk — a count-integrity test that re-trues on every merge. The manifest-plus-drift-check pattern is the same contract testing applied to operational documentation instead of lesson counts.

When a PR drops --legacy-mode from the deploy tool's accepted flags, the engineer updates the manifest in the same PR. The nightly drift-check job runs at most 24 hours later, finds every runbook that still references the removed flag, and fails with step-level detail:

DRIFT FAILURE: runbooks/deploy.md
  Step 3: flag '--legacy-mode' not found in manifest.flags['deploy-tool']
  Step 7: path './dist/legacy' not found in manifest.paths

Two actionable failures. Two runbook updates. Zero 3am surprises.

Scheduling the Verification Job

A PR-scoped trigger catches drift introduced by runbook edits but misses drift introduced by tooling changes in unrelated PRs. The deploy-tool upgrade that removed --legacy-mode did not touch any runbook files, so a PR-scoped trigger would not have caught it.

The complete schedule uses two triggers:

On PR (runbook-file changes only): Immediate feedback when a runbook is edited. Catches regressions introduced by the runbook author.

Nightly cron (all registered runbooks): Catches drift from tooling changes in unrelated PRs. The nightly window means a tooling change that breaks a runbook reference is surfaced within 24 hours, not within 45 days.

The nightly job runs against the full registry — every DocRecord with doc_type: 'runbook'. The registry from the doc-registry lesson is what makes "all registered runbooks" a concrete, enumerable set. Without the registry, the nightly job would need to know which files are runbooks by convention or glob pattern, which is itself a source of drift.

Converting Legacy Runbooks

Most existing runbooks were not written with drift testing in mind. Converting them to drift-testable form is straightforward:

1. Add code fences around shell commands so the parser can extract them cleanly.
2. Consolidate expected outputs into a dedicated Verify section.
3. Extract all referenced commands, flags, and paths and add them to the manifest.
4. Run the drift checker once against the current manifest — failures indicate runbook steps that reference something the manifest does not list, either because the manifest is incomplete or because the runbook is already stale.

The first pass through legacy runbooks often surfaces drift that has accumulated silently for months. Each failure is a future 3am incident that was caught while the stakes were low.

What's Next

The next lesson covers changelogs — the documentation type that is almost never written by hand on teams that adopt conventional commits. The same pipeline that generates runbook drift failures generates changelog prose: structured input (commits), mechanical classification (haiku), synthesis (sonnet), human gate before publish. A date-stamped changelog is also a machine interface: this academy reads lesson frontmatter dates to announce new content. The changelog is the same concept applied to the release event stream.