Debugging Multi-Agent & Orchestration Failures
Shared-state collisions, split-brain instances, the concurrent-agent git collision, and the score-component dead-component audit — failures unique to multi-agent systems.
The Failures That Single-Agent Debugging Misses
A single-agent system fails in predictable ways: the process crashes, the output is wrong, the logs show an error. Multi-agent systems can fail in ways that are invisible to any individual agent — because the failure lives in the coordination layer, not in the agents themselves.
The Agent Gateway concurrent-agent git collision. The Semantic Memory Layer :8080 split-brain. The Agent Framework 96.2% dead-component scoring system. Each of these produced valid-looking behavior in every individual component while producing no useful output from the system as a whole.
Debugging these failures requires a different starting point: inspect the shared state, not the individual agents.
The Four Multi-Agent Collision Patterns
These four patterns cover the dominant failure modes observed in Knox's multi-agent fleet:
Git working tree collision (concurrent-agent incident): Two agents share a working tree. Agent A edits a tracked file. Agent B runs git checkout -- . or git reset --hard as part of its workflow, discarding the uncommitted edits Agent A just made. The result is partially overwritten work and stray commits on unexpected branches.
Split-brain service instance (Semantic memory layer :8080): Two server instances bind to the same port via SO_REUSEPORT (a socket option that lets multiple processes listen on the same port simultaneously). Each serves approximately 50% of requests from divergent state. The symptom is intermittent, non-deterministic failures on identical inputs — which looks like model non-determinism until lsof reveals two processes.
Shared mutable context: Multiple agents read and write the same resource without coordination. The later write wins. Data from earlier agents is silently discarded. This is a classic race condition, and it manifests as mysteriously missing work rather than an explicit error.
Scoring component starvation (Agent framework): The components of a multi-component score have a ceiling that is architecturally below the threshold. The system processes all inputs correctly and blocks all of them. The pipeline runs but never fires.
The Split-Brain Anatomy
The split-brain case is worth examining in detail because it has a specific signature — 50% failure rate on identical inputs — that distinguishes it from all other failure modes.
A 5% failure rate means: occasional errors, likely transient. Retry logic helps. A 100% failure rate means: consistent bug, specific to this input or configuration. A 50% failure rate means: two instances, one correct, one not. No retry strategy fixes this — 50% of retries also fail.
The detection is always lsof -iTCP:<port>. If it shows two processes, split-brain is confirmed. The fix is to kill one instance and prevent the dual-launch from recurring (usually a launchd plist conflict with a Docker port mapping).
The Concurrent-Agent Git Collision
The Agent Gateway concurrent-agent collision happened because two Claude Code sessions were operating on the same repository checkout simultaneously. One was writing feature code; the other was writing tests. They were not in worktrees — they shared the same git HEAD.
Session 1 edits the tracked file src/feature.py. Session 2 runs git reset --hard to verify a clean base state — silently discarding all uncommitted changes to tracked files, including Session 1's edits. Session 1 continues working, not knowing its edits were just wiped.
The result: stray commits, missing changes, a git log that shows work from both sessions in unexpected order.
Prevention: one working tree per parallel agent. The rule is structural, not advisory.
# Before dispatching parallel agents:
git worktree add /tmp/agent-1-workspace -b agent-1-$(date +%s)
git worktree add /tmp/agent-2-workspace -b agent-2-$(date +%s)
# Each agent operates on its worktree path — never the shared working tree
The /parallel-agent-dispatch track covers worktree isolation in depth. The debugging lens here is: when parallel agent work produces partial or contradictory output, the first diagnostic is git worktree list — were they isolated or shared?
Fix the collision yourself: here is a dispatch script that points two parallel agents at the same checkout. Rewrite it so each agent operates on an isolated git worktree.
The Score-Component Dead-Component Audit
The Agent Framework 96.2% dead-component problem is a class of failure specific to scoring systems with multiple components. It is invisible to all standard monitoring because:
- The scoring function runs correctly
- The pipeline processes all inputs
- The logs show no errors
- The output (score values) looks reasonable
The only probe that reveals it: SELECT component_name, COUNT(*) as fires FROM score_events WHERE fire = TRUE AND created_at > NOW() - INTERVAL 30d GROUP BY component_name ORDER BY fires DESC.
When 7 out of 8 components return 0 fires over 30 days, the ceiling is the max value of the 1 live component. If that ceiling is below the threshold, the system can never trigger — regardless of market conditions, regardless of model quality, regardless of any per-signal tuning.
The fix is not to adjust the threshold. The ceiling is 3, the threshold is 25. Adjusting the threshold to 3 makes the system trigger on any signal regardless of quality. The fix is to audit each dead component: why is it firing at 0%? Was its data source disconnected? Was a configuration change set its weight to 0? Was the signal it monitors deprecated?
The Multi-Agent Debugging Protocol
When a multi-agent system produces wrong or missing output, use this sequence before looking at any individual agent's logs:
1. Check shared resources first. What does the working tree look like? How many processes are on the port? What is the queue depth? What is the score component distribution? These probes take seconds and rule out the entire class of coordination failures.
2. Verify isolation was actually used. For parallel agents dispatched with worktree isolation — confirm the worktrees exist and were used. git worktree list shows the current state. If worktrees are absent, shared-tree collision is the suspect.
3. Compare agent output counts. In a pipeline where Agent A feeds Agent B feeds Agent C, count the outputs at each handoff. If Agent A produced 100 items, Agent B processed 97, and Agent C produced 0 — the failure is between B and C. Look at the handoff contract.
4. For scoring systems, run the fire-rate audit. Pull component fire rates over the observation window. Any component at 0% is potentially dead. Compute the ceiling and compare to the threshold.
5. For intermittent failures, check the distribution first. 50% failure rate = split-brain. 5% = transient. 100% = systematic. The distribution tells you the fix class before you open any code.
Single-agent debugging tools are necessary but not sufficient for multi-agent systems. The shared state is where multi-agent bugs live — and only shared-state inspection can find them.