The Scientific Method for AI Debugging

What Failed Before Should Not Be Investigated Again

The Semantic Memory Layer split-brain incident took two debugging sessions to resolve. The first session disproved three hypotheses: ChromaDB index corruption (memory_reindex ran clean), network packet loss (direct curl succeeded), and API endpoint regression (the route existed and responded correctly).

The second session — days later, new context — started by reading the eliminated-hypotheses log. All three dead ends from session one were already marked DISPROVED with evidence. The session went directly to the remaining candidates and found the actual cause in forty minutes: two server instances (native launchd + Docker container) sharing port 8080 via SO_REUSEPORT — a socket option that lets multiple processes listen on the same port at once — and serving divergent state.

Without the log, session two would have re-investigated ChromaDB corruption. That is the cost of debugging without a hypothesis log: you pay for the same dead end multiple times.

The Six-Step Debugging Loop

The scientific method was designed for exactly the problem AI debugging faces: non-deterministic systems that cannot be reproduced reliably, where correlation and causation are easy to confuse.

Each step is a gate, not a guideline. The sequence matters:

Observe before hypothesizing. The exact failure artifact — the full error text, the specific log line, the precise output — is what enables a specific hypothesis. "The agent gave bad output" is not an observation. "The agent generated a summary of length 12 with no citations when the system prompt requires citations for all claims" is an observation. Specific observations produce specific hypotheses.

Hypothesize before instrumenting. The instrumentation must be designed to confirm or disprove the hypothesis — not to collect all available data. "Add logging everywhere" is not instrumentation. "Log the token count at the point where the system prompt is injected to verify it is not truncated" is instrumentation for a specific hypothesis.

Instrument before fixing. This is the most commonly violated step. The temptation to fix immediately — "I know what's wrong, I'll just change it" — is strong. But fixing without confirmation means you may have addressed a symptom. The instrumentation confirms causality. Without it, you cannot know whether the fix worked or whether something else changed.

Eliminate before continuing. Write the hypothesis to the log with its verdict and evidence before moving to the next candidate. The log is not a post-hoc record — it is written at the point of elimination.

The Eliminated-Hypotheses Log in Practice

Five real examples from Knox's production fleet, each showing the structure: hypothesis, evidence used to disprove it, and the actual root cause that was subsequently found.

Notice the pattern across all five examples: the initial hypothesis was plausible. ChromaDB corruption makes sense as a cause for Semantic Memory Layer 404s. A high threshold makes sense as a cause for zero trades. A rotated token makes sense for an auth failure. Each hypothesis was reasonable — and each was wrong. The log is what makes that wrongness available to future sessions without re-doing the investigation.

Writing Valid Hypotheses

A debugging hypothesis has three required properties:

It is specific. Not "maybe the context is too long" — "the agent drops the citation instruction when the message history exceeds 6,000 tokens because the system prompt is pushed to position 6,200 in the context window."

It is falsifiable. There exists a probe that can confirm or disprove it. "The model is getting worse" is not falsifiable — there is no single probe for that claim. "The model's output changes when the conversation is stripped to just system prompt + the failing user message" is falsifiable.

It predicts a specific mechanism. Not "something is wrong with the API" — "the API returns a 429 when the request rate exceeds 10 per second, and our pipeline sends 15 per second during load spikes." The mechanism tells you where to look and what to probe.

The /debug-investigate Skill

Knox's /debug-investigate skill codifies this loop as an automated capability. When triggered with a bug report, it:

1. Reads the eliminated-hypotheses log from Semantic Memory Layer memory for the relevant system
2. Generates 3 falsifiable hypotheses, ranked by cost to probe
3. Specifies the minimal instrumentation needed to confirm or disprove each
4. Records the verdict and evidence to the log on completion
5. Flags any hypothesis that has been disproved twice — indicating the assumed root cause space is wrong and the investigation needs to expand

The skill is the mechanism. The log is the value. The compound learning comes from the log persisting across sessions.

Ranking Hypotheses by Probe Cost

Not all hypotheses are equal in cost to test. When multiple candidates exist, rank them:

Cheapest probes first: pgrep -afl (lists running processes and their command lines), lsof -iTCP:8080 (lists the processes bound to a TCP port — two rows means two instances share it), SELECT COUNT(*) FROM artifacts WHERE created_at > NOW() - INTERVAL 1h. These take seconds and rule out entire hypothesis classes.

Medium-cost probes second: Running the system in isolation, stripping the conversation to minimal context, swapping one component. These take minutes.

Expensive probes last: Rebuilding containers, running full load tests, instrumenting at the database level. These take hours.

The Semantic Memory Layer split-brain would have been found in session one if lsof -iTCP:8080 had been run first. It showed two processes bound to the port in under a second. Instead, the session spent forty minutes on ChromaDB instrumentation before reaching the port check.

What the Log Is Not

The eliminated-hypotheses log is not a failure log. It is not a list of things that went wrong. It is a knowledge asset that encodes the structure of the problem space — specifically, which parts of that space are not the answer.

This distinction matters because it changes how the log is used. A failure log is read after an incident to understand what happened. The eliminated-hypotheses log is read at the start of every debugging session to understand where not to go. It is consulted before forming new hypotheses, not after finding the root cause.

The /debug-investigate skill reads the log before generating any hypotheses. The first question is not "what could cause this?" — it is "what have we already ruled out?" The log answers that question instantly, and the session begins in a smaller, better-defined search space.

That is the scientific method applied to AI debugging: systematic hypothesis elimination that makes the problem space smaller with every session, until the answer is the only thing left.