Ask Knox

The Complete Loop

Every lesson in this track has built toward a complete debugging discipline. The scientific method (the “The Scientific Method for AI Debugging” lesson), the gate chain (the “Reading the Right Signal” lesson), health vs. liveness (the “Health Is Not Liveness” lesson), context diagnosis (the “Debugging Context & Prompt Failures” lesson), multi-agent patterns (the “Debugging Multi-Agent & Orchestration Failures” lesson), and stop-and-replan (the previous lesson) are all components of one integrated playbook.

This lesson assembles the complete six-phase loop and adds the one phase that the other lessons only referenced: regression-test-first. The loop closes here.

The Six-Phase Playbook

The sequence is not flexible. Each phase is a gate — you do not proceed to the next phase until the current one is complete.

Phase 1 (Reproduce): Get the exact failure. The full error text. The specific input. The conditions. Confirm reproducibility: consistent or intermittent? Intermittent points to split-brain or non-determinism. Consistent points to a deterministic root cause.

Phase 2 (Hypothesize): Write 2-3 falsifiable hypotheses. Check the eliminated-hypotheses log first — has this been disproved before? Rank by cost to probe. The cheapest probe that rules out the most hypothesis space goes first.

Phase 3 (Instrument): Pull the gate chain. Check downstream artifact count. For AI: test fresh session behavior. For multi-agent: inspect shared state (lsof, git worktree list, queue depth). No fixes yet.

Phase 4 (Eliminate): Apply the 2-attempt rule. Write each verdict to the eliminated-hypotheses log. Communicate pivots explicitly. Gate triggers if two attempts fail in the same direction.

Phase 5 (Fix): Minimal fix for the confirmed root cause. No "while I'm here" changes. For production server services: verify on the production server, not your laptop localhost. For multi-agent: verify on the actual system, not an isolated component.

Phase 6 (Regression-Test): Write the test before the fix. Commit fix and test together. Add to CI. This phase is not complete until CI runs the test and it passes.

The Regression-Test-First Discipline

The sequence is: (1) write a test that fails against the current code, (2) apply the fix, (3) verify the test now passes, (4) commit fix + test together.

Step 1 is the one most often skipped. The test before the fix feels redundant — you already know the bug. But it serves a purpose that the after-fix test does not: it confirms the test actually captures the bug, not just a passing behavior.

# Example: Foresight position sizing bug
# Step 1: Write the failing test FIRST

def test_position_size_not_10x():
    """Regression test for 10x sizing bug — must FAIL before fix."""
    calculator = PositionCalculator(balance=1000, risk_pct=0.01)
    size = calculator.calculate(market="BTC-USD", price=50000)
    
    # With the bug: size = 100 (correct formula × 10x multiplier bug)
    # After fix: size = 10 (correct: 1000 × 0.01 = $10 position)
    assert size == pytest.approx(10.0, rel=0.01), f"Expected $10, got ${size}"

# This test FAILS with the current buggy code — that's correct.
# Apply the fix. The test now passes.
# Commit both together.

The test that fails before the fix is a live specification of what correct behavior looks like. It is not documentation — it is an executable contract that CI will enforce on every future run.

The Laptop-vs-Production-Server Trap in the Playbook

Phase 5 (Fix) has a specific failure mode in distributed infrastructure: verifying on your laptop localhost when the production system is the production server.

The April 7 incident (detailed in the “Why AI Systems Fail Differently” lesson context): eight PRs were applied, all verified on laptop localhost. The fix worked on the shadow stack. Production broke immediately when the Vercel frontend (pointed at the production server) hit the unchanged production server backend.

In the playbook, phase 5 is not complete until the fix is on the production system:

# Phase 5 for backend services — not done until:
ssh 198.51.100.5 'cd ~/dev/<repo> && git pull && \
    export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH && \
    cd ~/dev && docker compose build <service> && \
    docker compose up -d <service>'

# Then verify against the production endpoint:
curl https://api.example.app/<endpoint>
# NOT: curl http://localhost:8001/<endpoint>

Your laptop localhost verifies the shadow stack. The production server verifies production. Both checks are required for a backend fix to be in phase 5.

The Audit Lab

The playbook generates two durable artifacts: the eliminated-hypotheses log and the regression test suite. Together, these are the "audit lab" — a compound knowledge base that makes AI debugging cheaper with every bug fixed.

The eliminated-hypotheses log grows: After 50 debugging sessions, the log contains 150+ disproved hypotheses. Each new debugging session starts with a narrower search space. The system becomes easier to debug over time, not harder.

The regression test suite grows: After 100 bugs fixed with regression tests, the CI suite has 100 guards. A recurrence of any of those bugs is caught in seconds, not discovered in production hours later.

The compound effect is measurable. Knox's fleet now has 1,970+ tests in Foresight, 414 tests in Semantic Memory Layer, 431 tests in the academy. Each of those tests is a closed loop — a bug that was found, understood, fixed, and permanently guarded.

Applying the Playbook to the Agent Framework Case

Retrospectively, the Agent Framework case walked through all six phases — eventually:

Reproduce: 0 trades over 21 days. Confirmed consistent, not intermittent.

Hypothesize: (1) Threshold too high. (2) Signal quality degraded. (3) Exchange API connectivity. Eliminated-hypotheses check: threshold checked before (no log entry) — will re-investigate.

Instrument: Pull distribution. SELECT block_reason, COUNT(*) FROM signal_decisions GROUP BY block_reason. Results: all 2,646 signals blocked at threshold gate with score < 40%. Add: SELECT component, AVG(score), COUNT(*) FROM component_scores GROUP BY component.

Eliminate: Threshold hypothesis — DISPROVED. Score values never approached 40% regardless of signal quality. Root cause was ceiling, not threshold value. Score-component audit: 7 of 8 components at 0% fire rate. Ceiling = 3.8%.

Fix: Reactivate 4 dead components. Rebalance weights via arithmetic backtest. Verify ceiling now > 40%.

Regression-test: test_score_ceiling_above_threshold() — asserts that the ceiling (sum of max values from firing components) exceeds the threshold by at least 10%. This test would catch any future component death that drops the ceiling below threshold.

The fix took one session once the diagnostic path was clear. The missing element in the real incident was the score-component audit — which took three weeks to reach because the wrong modules were investigated first. The playbook would have pulled the component fire rates in phase 3.