ASK KNOX
beta
Learn/Documentation is King/Lesson 545
LESSON 545

Docs as Contracts: References That Don't Drift

Hand-written API docs drift the moment schema and doc evolve independently — generate from a single source or pin with drift tests.

8 min read·Documentation is King

When a Doc Becomes a Lie

An API reference that was accurate on day zero can actively mislead a consumer by week ten. Not through any act of malice — through a completely ordinary sequence: a field gets renamed, the TypeScript type is updated, the tests pass, the PR is merged, and the API docs are quietly never touched.

The code is correct. The doc is wrong. And a consumer relying on that doc — especially an agent that reads it at the start of every session — will build against a contract that no longer exists.

This is the most dangerous class of documentation problem: not a missing doc, but a confidently wrong one. A missing doc forces someone to ask. A stale doc makes them confident about the wrong thing.

The drift happens in stages. Day zero: schema and doc are in sync, consumers trust both. Week two: the schema is updated, the code compiles, but docs are not reviewed in the PR. Week six: two new consumers have onboarded using the stale doc and built against the wrong field name. Week ten: an agent implementing a new feature reads the stale doc and generates incorrect code.

None of these steps required negligence. They required only the absence of an enforcement mechanism.

The Single Source Problem

The root cause of contract drift is almost always the same: two things need to stay in sync, but there is nothing that mechanically keeps them in sync.

The classic pattern is the hand-written type that mirrors a schema:

// schema/lesson-response.json says:
// { lesson: number, title: string, track: string, excerpt: string }

// types/lesson-response.ts says:
export interface LessonResponse {
  lesson: number
  title: string
  track: string
  excerpt: string
}

These start identical. They will diverge. The question is when, not if. Both are sources of truth. There is no owner.

The fix is to eliminate the duplication by establishing a single source and making everything else a derived output.

With a generation pipeline, the JSON Schema is the owner. The TypeScript type is generated from it on every CI run. The docs are generated from it. When the schema changes, both the type and the docs change in the same commit. There is no manual sync step that someone can forget.

This is the same principle behind every good abstraction in software: one source of truth, multiple derived representations. The schema is the source. Types and docs are representations.

When You Cannot Generate

Sometimes generation is not practical — the API is external, the schema belongs to another team, or the integration is read-only and you cannot run a generator. In these cases, the second-best option is pinning the contract with a drift test.

A drift test does not verify that your implementation is correct. It verifies that the interface has not changed from what you documented. It is a canary, not a proof.

// tests/drift.test.ts
import { describe, it, expect } from "vitest"
import type { LessonResponse } from "../types/lesson-response"
import schema from "../schema/lesson-response.json"

describe("LessonResponse contract drift", () => {
  it("schema required fields all exist on the TypeScript type", () => {
    const sample: LessonResponse = {
      lesson: 545,
      title: "Docs as Contracts",
      track: "documentation-is-king",
      excerpt: "Hand-written contract docs drift.",
    }

    schema.required.forEach((field: string) => {
      expect(field in sample).toBe(true)
    })
  })
})

The test is simple: build a sample object that satisfies the TypeScript type, then assert that every required field from the schema is present in it. If someone renames title to lessonTitle in the type without updating the schema, the sample will not have a title key and the test fails.

This test runs in CI. A renamed field fails the build before it reaches consumers. That is the enforcement mechanism that convention cannot provide.

Consumer-Contract Thinking

The drift test is the last line of defense. The first line is the question you ask before changing an interface: who consumes this, and what shape do they expect?

This is the consumer contracts discipline from the spec-driven-development track, applied at the documentation layer. Before changing an API response shape, before renaming a field, before making an output optional that was required — grep the codebase for callers.

# Who calls this API or consumes this type?
grep -rn "LessonResponse\|lesson-response" $(git ls-files)

The result is your consumer list. Every item on that list is a file that will break if you change the shape without migrating callers. Write that list down. If there is a doc that consumers rely on, that doc is part of the contract. Updating it is not courtesy — it is part of the change.

This is what makes the consumer contract section in a PRD so powerful: it forces the research step before the change is made, not after consumers start filing bugs.

The Academy Example

This academy's content integrity test in __tests__/content-integrity.test.ts is a drift test in exactly this sense. It asserts that the number of lessons listed in lib/tracks.ts matches the number of MDX files in content/academy/. When a new lesson is added, the test catches a missing tracks.ts registration before the PR is merged.

That test is not clever. It does not validate content quality. It pins a structural contract — the invariant that every lesson file has a corresponding tracks.ts entry — and fails loudly when that invariant is violated.

Every codebase has invariants worth pinning. The question is whether they are pinned in tests or enforced by hope.

Immutable vs Living vs Generated

One more distinction worth naming. The docs taxonomy from the “The Documentation Stack” lesson classifies docs as immutable, living, or generated:

  • Immutable: ADRs. Once accepted, they are never edited — they are superseded by a new ADR. The contract is preserved.
  • Living: READMEs, runbooks, lessons files. They are maintained by convention, which means they benefit most from CI freshness gates.
  • Generated: Types from schemas, API docs from OpenAPI specs. The generation pipeline makes drift impossible by construction.

Contract docs — API references, schema docs, type definitions — should be generated wherever possible. When they cannot be generated, they should be pinned by drift tests. When they are neither generated nor pinned, they will drift. The only question is how long until they mislead someone.

Now build the enforcement mechanism: a JSON Schema for an API response, and the drift test that pins consumer code to it.