The Doc-Bot Safety Model

The Helpful Bot That Hallucinated

A doc-updating bot was given access to a repository's architecture document and asked to refresh it after a significant refactor. The bot analyzed the diff, read the existing doc, and produced a confident, well-structured update describing the new caching strategy in detail — including a configuration option called cache.invalidation_strategy that controlled whether invalidation was lazy or eager.

The option had never existed. The bot had inferred it from the code structure — plausibly, coherently, incorrectly. The PR passed the freshness gate. The generated content carried today's timestamp. Two engineers and one agent session later tried to implement against the config option before anyone noticed the doc described behavior that had never shipped.

The failure mode was not that the bot lied. The failure mode was that the bot's output was indistinguishable from correct output, carried a fresh timestamp that signaled recent verification, and entered the codebase without a human reading it for accuracy.

This is the hallucinated-docs failure mode: plausible, well-written, and wrong. It is worse than a stale doc for one reason: recency is a trust signal.

The Mechanical/Semantic Boundary

The safety model rests on one distinction: whether the change is mechanical or semantic.

Mechanical changes are changes that can be verified by a deterministic check. A count updated to match the actual count. A filename in a table replaced with the new filename. A link corrected to point to the live URL. A version string bumped to match the release tag. These changes are correct or incorrect by an objective, computable measure — run the check, compare the result, done.

Semantic changes are changes where the content describes system behavior, architecture, or procedures that a reader will act on. An architecture doc updated to reflect the new service topology. A runbook step rewritten with the new flag syntax. A hard rule added to the context file. An ADR rationale section revised. These changes cannot be verified by running a deterministic check — they require someone who understands the system to read them and confirm they are accurate.

The boundary determines the routing decision. Mechanical changes that pass the freshness gate and other CI checks can auto-merge safely — the bot's output is either right (matches the deterministic check) or wrong (fails the check). Semantic changes always route to human review, with no exceptions.

The "when in doubt" rule is always: classify as semantic. The cost of misclassifying a semantic change as mechanical is a potential hallucination in the live docs. The cost of misclassifying a mechanical change as semantic is one extra PR review. The asymmetry is decisive.

The Change Classifier

The classifier is a claude-haiku-4-5 call with structured output. It receives the diff text and a system prompt that defines the mechanical/semantic boundary precisely. The output schema has three fields: change_class (the routing decision), rationale (one sentence), and sections_touched (which parts of the doc changed).

The schema is validator-legal: every object carries additionalProperties: false, no minItems/maxItems/minLength/maxLength/minimum/maximum constraints.

const CLASSIFICATION_SCHEMA = {
  type: "object",
  additionalProperties: false,
  properties: {
    change_class: { type: "string", description: "mechanical or semantic" },
    rationale: { type: "string", description: "one sentence explaining the classification" },
    sections_touched: {
      type: "array",
      description: "list of section headings or field names changed",
      items: { type: "string" },
    },
  },
  required: ["change_class", "rationale", "sections_touched"],
}

After the classifier returns, the code applies a safe-default rule: treat any change_class that is not exactly the string "mechanical" as semantic. This handles parse failures, unexpected model output, and any edge case where the classifier's output is ambiguous — all of them route to human review.

const changeClass: "mechanical" | "semantic" =
  parsed.change_class === "mechanical" ? "mechanical" : "semantic"

The safe default is the entire safety model in one line. Build defaults toward caution.

Provenance Footers

Every bot-generated PR carries a provenance footer — a structured block appended to the PR description that records the identity of the system that made the change, the input that triggered it, the classification result, and the scope of what was touched.

The provenance footer serves three purposes. First, it makes bot edits distinguishable from human edits at a glance — reviewers immediately know they are reading generated content. Second, it makes the classification decision auditable — if a change that should have been semantic was classified as mechanical, the footer shows exactly what classification was made and what rationale was given. Third, it makes the trigger traceable — what merge or event initiated the bot run, and what diff it analyzed.

Without the provenance footer, a bot-generated PR is indistinguishable from a human PR until something goes wrong. After something goes wrong, reconstructing the chain — what triggered the bot, what diff it read, what it classified — requires searching merge logs, CI records, and API call history. The footer captures all of this at generation time, where the cost is zero.

The provenance footer is not optional for semantic changes routed to human review — it is especially important there, because it is what the reviewer reads to understand what they are being asked to approve.

The Complete Safety Loop

The mechanical/semantic boundary, the safe-default classifier, and the provenance footer together form the safety loop:

1. Bot analyzes the diff and identifies affected docs via the registry
2. Bot drafts the doc patch — mechanical sections updated, semantic sections flagged for human review
3. Classifier runs on the diff and produces a routing decision
4. PR opened with provenance footer attached
5. CI runs freshness gate and other checks
6. Mechanical PRs with passing gates can auto-merge; semantic PRs sit in the human review queue
7. Human reviewer reads the provenance footer, understands the scope, approves or rejects

The critical invariant: the bot never pushes to main. Every bot change goes through a PR. The PR is the enforcement point for the routing decision — a mechanical PR that somehow had a semantic change would still have a human reviewer if the CI classification was wrong, because the provenance footer makes the classification visible and auditable.

This is how a trading-system repo's operational documentation stays current without human maintenance sessions: mechanical updates (agent version strings, count tables, link references) auto-merge when CI passes; architecture changes and runbook revisions sit in the review queue until a human with system knowledge approves them.