Ask Knox

The Bot That Was Working Fine

Three weeks. Two thousand, six hundred and forty-six signals generated. Zero trades placed.

Every monitoring check passed. The process was alive. The logs were writing. The heartbeat was green. The scoring pipeline was running. By every liveness metric, Agent Framework was working perfectly.

It was not working at all.

This is the central problem of debugging AI systems. They fail differently from traditional software — not with crashes and stack traces, but with silence, with ghost signals, with outputs that look correct until you check whether any useful work was actually done. The Agent Framework incident is not an edge case. It is the prototype of how AI systems fail.

Four Failure Classes That Don't Exist in Traditional Software

Traditional software debugging operates on a simple contract: same input, same output, deterministic stack trace when it breaks. AI systems violate all three.

These four classes require fundamentally different debugging strategies than the ones taught for traditional software:

Non-determinism means you cannot rely on reproduction. A bug that manifests 30% of the time requires statistical analysis, not a single reproduce-and-fix cycle. You are looking for distributions, not events.

Silent degradation means your monitoring will lie to you. A green dashboard combined with zero downstream output is the zombie pipeline signature. You must instrument artifact production, not process liveness.

Context and prompt failures account for the vast majority of "the model got worse" reports. Before suspecting the model, check the system prompt (git diff since last-known-good), check the conversation history (did prior turns pollute the in-context state?), and run a fresh session with the same prompt. If the fresh session works, context is the culprit.

Infrastructure failures in AI systems have a specific flavor: split-brain service instances, where two servers compete on the same port and serve divergent state. This produces intermittent, apparently random failures that no individual instance would show in isolation.

The Determinism Spectrum

Not all AI systems fail equally. The degree of non-determinism scales with the number of AI components in the pipeline.

Traditional software sits at 100% reproducibility. A multi-agent swarm may be effectively unreproducible — N agents, N orderings, shared mutable state. The debugging strategy must match the tier.

For high-reproducibility systems (traditional code, deterministic pipelines), the standard reproduce-isolate-fix approach works. For low-reproducibility systems (AI agents, multi-model pipelines), you need hypothesis-driven debugging with statistical confirmation. A bug is "fixed" when the success rate across 10 runs meets the expected distribution — not when one run passes.

Why This Track Exists

Most debugging education was built for deterministic systems. The heuristics it teaches — reproduce the bug, read the stack trace, fix the line that threw — fail for AI pipelines in the same way that a voltmeter fails to diagnose a structural problem with a bridge. It is the right tool for the wrong system.

Knox's infrastructure runs Agent Gateway (24/7 agent orchestration), Semantic Memory Layer (semantic memory at :8080), Foresight (Polymarket trading), Sports Prediction Agent (sports markets), and a perps trading bot (Phemex futures) — all AI-adjacent, all running continuously, all capable of silent degradation. The debugging patterns in this track are drawn directly from incidents in that fleet.

The agent framework that ran 2,646 signals and zero trades for three weeks. The Semantic Memory Layer instance that produced random 404s because two servers were sharing a port via SO_REUSEPORT and serving divergent ChromaDB state. The Foresight bot that appeared idle when the wallet balance had been withdrawn (soft-shutdown pattern). The perps bot that generated orders but never submitted them because the USDT futures wallet was empty while a different wallet showed a balance.

Each of these required a debugging approach that standard monitoring and traditional debugging techniques completely missed.

The Debugging Contract for AI Systems

The standard debugging contract — reproduce, isolate, fix — needs three amendments for AI systems:

1. Replace "reproduce" with "characterize the distribution." For non-deterministic failures, ask: how often does this fail, under what conditions, with what outputs? Distribution beats single-instance reproduction.

2. Replace "read the stack trace" with "pull the gate chain." AI pipelines have gates — market filter, direction check, momentum gate, order submission. The logged error names the gate that blocked one request; the root cause is often a different gate upstream that blocks most requests before they reach it. Read the gate chain plus the block distribution to find the module to fix.

3. Replace "fix the line that threw" with "prove downstream artifact growth." A fix that makes the process run without producing output is not a fix. The terminal test is artifact count: trades placed, rows written, decisions executed.

These amendments are the foundation for everything in this track. The next lesson builds the systematic hypothesis-driven loop that applies them.

What Comes Next

This track walks through the complete AI debugging discipline:

The next lesson: The scientific method applied to AI systems — hypothesis-driven loops, the eliminated-hypotheses log, and why preventing re-investigation of dead ends is a compounding asset.
The “Reading the Right Signal” lesson: Reading the right signal — gate chains, log interpretation, and the difference between visible failure and dominant failure.
The “Health Is Not Liveness” lesson: Health vs. liveness — proving downstream artifact growth, the zombie pipeline pattern in detail, and how to build health checks that don't lie.
The “Debugging Context & Prompt Failures” lesson: Debugging context and prompt failures — the most common AI bug class, almost always misdiagnosed as a model problem.
The “Debugging Multi-Agent & Orchestration Failures” lesson: Multi-agent debugging — shared state collisions, split-brain diagnosis, and the scoring system dead-component audit.
The “The Stop-and-Replan Discipline” lesson: Stop-and-replan discipline — the 2-attempt rule, the incidents that happen when it is violated, and how to communicate a pivot.
The “The AI Debugging Playbook” lesson: The complete AI debugging playbook — the full Reproduce → Hypothesize → Instrument → Eliminate → Fix → Regression-Test sequence.

The Agent Framework bot was eventually fixed. The root cause: 96.2% of scoring components had a fire rate of zero. The ceiling was 3.8%. The threshold was 40%. Mathematically unreachable. The fix was not to adjust the threshold — it was to audit and rebalance the dead components. That diagnosis took a score-component audit, not a liveness check.