// Track

Debugging AI Systems

Root-Cause Discipline for Agents & Pipelines

The debugging discipline built from real incidents in Knox's AI fleet — zombie pipelines, split-brain instances, concurrent-agent git collisions, dead scoring components, and context pollution. Scientific method, hypothesis logs, gate chains, and the regression-test-first rule that makes debugging compound over time.

Recommended: Complete Parallel Agent Dispatch & Worktrees first

8 lessons~71 min total

Lessons are shown in recommended order. Complete them in sequence for the best experience — or jump to any lesson.

Lesson 473·8 min read

Why AI Systems Fail Differently

Non-determinism, silent degradation, and the 'looks like it works' trap — the four failure modes that make AI debugging a distinct discipline.

Lesson 474·9 min read

The Scientific Method for AI Debugging

Hypothesis-driven debugging with a persistent eliminated-hypotheses log — the compound-interest approach that prevents re-investigating the same dead ends across sessions.

Lesson 475·9 min read

Reading the Right Signal — Logs, Traces & Gate Chains

Pull the gate chain before proposing a fix — the visible failure and the actual block are rarely at the same gate, and fixing the wrong module wastes hours.

Lesson 476·8 min read

Health Is Not Liveness

A process can be alive and a zombie simultaneously — heartbeat checks confirm the process exists, not that work is flowing. Prove downstream artifact growth.

Lesson 477·9 min read

Debugging Context & Prompt Failures

Bad output is almost always a context or prompt problem — not the model. The reread/pollution lens diagnoses 90% of 'the model got worse' reports in minutes.

Lesson 478·10 min read

Debugging Multi-Agent & Orchestration Failures

Shared-state collisions, split-brain instances, the concurrent-agent git collision, and the score-component dead-component audit — failures unique to multi-agent systems.

Lesson 479·8 min read

The Stop-and-Replan Discipline

The 2-attempt rule: two failed attempts in the same direction mean the assumption is wrong, not the execution. Stop, re-examine, communicate the pivot before attempting again.

Lesson 480·10 min read

The AI Debugging Playbook

The complete six-phase playbook: Reproduce → Hypothesize → Instrument → Eliminate → Fix → Regression-Test. The audit lab that makes AI debugging compound over time.