Runbooks: Docs That Execute Under Pressure

Tribal Knowledge Fails at 3am

The trading bot started returning 502 errors at 2:47am. The on-call engineer was paged. They knew, in general, how to restart the service — but not the exact command, not the expected output that confirmed a clean restart, not whether to check the database before or after, and not what rollback looked like if the restart failed.

Twenty-three minutes of Slack messages to the original engineer, who was asleep. A restart performed from memory, with slightly wrong flags. A 15-minute window of further downtime while the wrong restart was diagnosed and corrected.

The total outage: 41 minutes. The fix, once properly executed: 3 minutes. The 38-minute difference was the cost of tribal knowledge — knowledge that lived in one engineer's head and was unavailable at 2:47am.

What Makes a Runbook Different from a Script

A script executes. A runbook is executable by a person. The difference matters.

A script assumes all preconditions are met. A runbook verifies preconditions before touching anything. A script produces machine-readable output. A runbook documents expected human-readable output for each step so the operator knows whether to proceed or stop. A script has no rollback — it either succeeds or it doesn't. A runbook has an explicit rollback that can be executed by anyone who can follow numbered steps.

The six-section anatomy in the diagram is not arbitrary. Each section exists because something went wrong in production when it was missing.

Preconditions exist because steps 1–4 often assume state that is not always true: SSH access, a clean git status, enough disk space, a database that is reachable. Verifying preconditions before step 1 catches these conditions before they turn into a halfway-executed runbook that is harder to recover from than the original problem.

Expected output per step exists because operators under pressure cannot distinguish "normal output I don't recognize" from "error output I should stop for." When the runbook says "Expected: Already up to date." and the operator sees a list of changed files, they know the state is different from expected and can decide whether to proceed.

Verification exists because "service is running" is not the same as "correct version is serving." The verification section proves the artifact changed — the right version number, the right row count in the database, the right response from the endpoint. A liveness check passes for any version of the code. Artifact-based verification is specific to the change you just deployed.

Rollback exists because it cannot be written well under pressure. The engineer who wrote the runbook is the person who knows what changed, in what order, and how to undo it cleanly. That is the person who should write the rollback section — before the incident, not during it.

The Two-Machine Lesson

The most common runbook gap in engineer environments with both a development machine and a production machine: the runbook assumes that "deploy" means "push code." It does not.

A deploy to a production machine has two independent steps:

1. Pushing the code to the version control system (from the dev machine)
2. Pulling the code onto the production machine (on the production machine)

A developer who verifies the deploy on their laptop has verified step 1. Step 2 — pulling the new code onto the production machine and restarting the service — must be verified independently. The production machine has its own git checkout, its own running processes, and its own environment variables. These are not automatically updated when the developer pushes.

The runbook for a production deploy must include the production-side steps: SSH to the production machine, git pull, confirm the expected commit is checked out, npm ci to install from lockfile, restart the service, verify the version endpoint returns the new version. None of these steps are redundant with the developer's local verification.

This pattern appears in Foresight deployments: the development machine confirms the code is correct; the production machine's runbook confirms the code actually landed. Both checks are necessary. Neither is sufficient alone.

Verification Beats Liveness

The most underbuilt section of most runbooks is the Verify section. Teams write preconditions and steps, then verify by checking if the process is running. This is not verification.

Consider a deploy where the new binary was built incorrectly and exits immediately on startup. The process manager restarts it automatically. From a liveness perspective — "is the process running?" — the answer is yes. From an artifact perspective — "is the correct version serving?" — the answer is no.

The correct verification pattern:

# Verify: artifact-based, not liveness-based
curl -s http://localhost:3001/api/version
# Expected: { "version": "2.1.0", "status": "ok" }

sqlite3 data/trades.db "SELECT count(*) FROM trades WHERE status = 'active';"
# Expected: same count as pre-deploy (confirm active positions survived)

These two checks prove that: (1) the new version is actually serving responses, not just running as a process, and (2) the data that matters survived the restart intact. Neither check can be faked by a crashed-and-restarted binary.

The rule: the verify section must check an artifact that only the new code can produce. If the check could pass with the old code deployed, it is not a verification.

Runbooks for Agents

Runbooks are not just for 3am human incidents. They are also the operational context that a coding agent needs to understand how a system is deployed and maintained.

When Knox dispatches an agent to update the Foresight trading bot, the agent needs to know: what is the deploy process? What are the preconditions? What does a successful deploy look like? An agent without this context will infer a deploy process from the codebase — and it will probably be correct 80% of the time, wrong 20% of the time, and the 20% will hit production at the worst moment.

A runbook loaded into the agent's context at dispatch time changes the probability distribution. The agent does not need to infer the deploy process — it reads it. The preconditions are explicit. The verification steps are defined. The rollback is documented.

For Tesseract Intelligence's trading bots, the runbook is part of the key-files table in the context file. Any agent session that might touch the deploy process loads the runbook as part of session initialization.

Converting Tribal Knowledge

Most teams have tribal knowledge but not runbooks. The conversion is simpler than it looks. The pattern:

1. Ask the person who knows to narrate the process once, out loud or in writing — without structure.
2. Extract the structure: what had to be true first? What commands were run? In what order? What did success look like after each step?
3. Write the preconditions from the "what had to be true first" extraction.
4. Number the steps and add the expected output for each.
5. Write the verify section by asking: "how would I prove this worked to a stranger who wasn't watching?"
6. Write the rollback by asking: "if step 4 failed, how do I get back to the state before I started?"

The first runbook is the hardest. The second is much easier, because you have a template.

Now write one. The challenge below gives you a tribal-knowledge deploy narrative and asks you to produce the runbook.

What's Next

The next lesson covers the learning loop — lessons files and post-mortems. Where runbooks document how to do things right, lessons files document how things went wrong and extract the rules that prevent repetition. Together, ADRs, runbooks, and lessons files form the operational memory of a system: the decisions made, the operations understood, and the mistakes not repeated.