Ask Knox

The Quiet Killers

The most dangerous production failures for MCP servers are not the ones that crash the process. Those are loud — the supervisor logs them, the monitoring alerts on them, the operator fixes them. The dangerous failures are the ones that let the server return 200 while producing wrong or stale results.

Knox's Semantic Memory Layer experienced three of these in production:

The split-brain incident: two server instances on port 8080 via SO_REUSEPORT, each serving divergent ChromaDB state. memory_query returned different results on subsequent calls. The server appeared healthy. The error was non-deterministic and took three days to diagnose.
The ingest-corpus stoppage: the cron job that adds lesson MDX to the index stopped running. memory_query continued returning results — just three-week-old results. The GET /health check returned 200 throughout.
The silent write failure: memory_remember returned 200 but the ChromaDB embedding step failed. The memory was logged as stored but never appeared in search results. The write-path was broken while the server appeared functional.

All three were caught by the observability patterns in this lesson. All three would have been missed by a naive GET /health check.

The Split-Brain Incident

The split-brain incident happened on the production server when both Docker and a legacy launchd plist started Semantic Memory Layer on port 8080. SO_REUSEPORT — a socket option that allows multiple processes to bind the same port — let both succeed without error.

From that moment, the OS load-balanced incoming requests between the two instances. A memory_remember call might go to the Docker instance. A subsequent memory_recall might go to the launchd instance. The Docker instance had a different ChromaDB client than the launchd instance — they served divergent state.

The symptom that surfaced it: memory_query("memory-service architecture") returned different results on three consecutive calls in the same Claude Code session. The session was not changing. The queries were identical. The results varied because different instances were answering.

Detection: lsof -i -P | grep 8080. Two PIDs. Fix: launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.example.memory-service.plist. Then docker compose restart memory-service. Then verify one PID.

The permanent fix was deleting the launchd plist — not just killing the process. KeepAlive=true would have restarted it.

The Health Check Hierarchy

The three-level health check hierarchy is not theoretical. Each level catches failure modes the level below misses. The false-healthy scenarios table in the diagram above shows this concretely — scenarios where L1 passes and L3 fails are exactly the scenarios that caused Knox's production incidents.

The L3 health check implementation for Semantic Memory Layer:

app.get('/health', async (req, res) => {
  const status = await collection.count()
  const lastIndexed = await getLastIndexedAt() // from SQLite sidecar

  const isHealthy = status > EXPECTED_MIN_CHUNKS &&
    Date.now() - lastIndexed < 24 * 60 * 60 * 1000 // 24 hours

  res.status(isHealthy ? 200 : 503).json({
    status: isHealthy ? 'ok' : 'degraded',
    chunks_indexed: status,
    expected_min_chunks: EXPECTED_MIN_CHUNKS,
    last_indexed_at: new Date(lastIndexed).toISOString(),
    age_hours: Math.round((Date.now() - lastIndexed) / 3600000),
  })
})

The 503 when unhealthy is intentional. Supervisors that poll /health and respect HTTP status codes will stop sending traffic to a degraded instance — or trigger a restart. A 200 from a degraded instance hides the problem.

Structured Logging

The observability stack diagram shows what to log at each level and — critically — what never to log. The "never log" column is the production rule that prevents secret leaks through the log pipeline.

For Semantic Memory Layer, the production log format is:

{
  "level": "info",
  "timestamp": "2026-05-31T03:14:26.341Z",
  "caller_id": "claude-code-read",
  "tool_name": "memory_query",
  "namespace": "project-docs",
  "results_count": 8,
  "duration_ms": 142,
  "query_truncated": true
}

Notice what is not in the log: the query string (may contain sensitive information), the bearer token (never), the full results (too large), or the raw request body (contains auth headers). The log contains enough to debug performance issues and audit access patterns, but nothing that could be sensitive.

Put both halves together in code: replace a free-text log and a liveness-only health check with a structured log line and the L3 check that proves the index is actually fresh.

Read-After-Write in Practice

The read-after-write verification for memory_remember:

async function rememberAndVerify(content: string, namespace: string) {
  const { memory_id } = await memoryRemember(content, namespace)

  // Verify the write landed
  const recalled = await memoryRecall(memory_id)
  if (!recalled) {
    logger.error({
      event: 'write_verification_failed',
      memory_id,
      namespace,
    })
    return { ok: false, error: 'Write succeeded but recall returned 404 — index inconsistency', code: 'WRITE_VERIFY_FAILED' }
  }

  return { ok: true, memory_id }
}

This pattern is the production standard for all write operations. The 200 from the write is not the success signal — the 200 from the read is. Any gap between the write response and the read result indicates a broken write path that must be surfaced before the caller gets a success response.

What's Next