Ask Knox

The Incident

On March 29, 2026, a debugging session exposed one of the more disorienting failure modes in distributed infrastructure: the same port on the same machine returning completely different data depending on how you connected to it.

The system under investigation was Akashic Records, the knowledge indexing service running on Knox (Mac Mini). Here is what the raw numbers looked like:

Two requests, one machine, one port. Different chunk counts. Different index states. The service returning 2,817 chunks was healthy — it was the one running under Docker, bound to the loopback interface. The service returning 4,520 chunks was a stale instance from an earlier test session, running under a different process with a different data volume mounted.

The MCP server in Claude Code's configuration pointed to 100.91.193.23:8002 — the Tailscale IP. Every knowledge query during that session silently hit the stale instance.

How Networking Creates This Ambiguity

To understand why this happens, you need a model of how services bind to network interfaces.

When a server starts, it listens on a combination of IP address and port. The IP address controls which network interface accepts incoming packets.

— The virtual interface 127.0.0.1 (or ::1 for IPv6). Only processes on the same machine can connect. A service bound here is invisible to all external traffic, including Tailscale.

— When a service binds to 0.0.0.0, it accepts connections on every interface: loopback, LAN, Tailscale, VPN. This is the typical Docker default when you publish a port.

— A service can bind exclusively to 192.168.1.x (LAN) or 100.x.x.x (Tailscale). Traffic from other interfaces is rejected at the OS level, regardless of port.

The table below shows what a machine with multiple network interfaces looks like from the outside:

How you connect	Interface used	What you hit
`curl localhost:8002`	`127.0.0.1` loopback	Process A (Docker, port published to loopback)
`curl 192.168.1.x:8002`	LAN adapter	Process B (if it exists) or nothing
`curl 100.91.193.23:8002`	Tailscale adapter	Process C (if it exists) or nothing

If two processes happen to be serving on port 8002 — one on loopback, one on a Tailscale-visible interface — you get exactly the March 29 scenario: the same port, completely different services.

Why Testing Locally Masked the Problem

The classic debugging mistake in this incident was testing with the wrong URL.

When investigating why Claude Code's knowledge queries seemed stale, the initial check was:

curl localhost:8002/discover

This returned a healthy response with 2,817 chunks and a proper list of namespaces. The service appeared fine. The investigation moved elsewhere.

What should have been run first:

curl http://100.91.193.23:8002/discover

That URL — the exact URL the MCP client was configured to use — was the stale instance. The 7-minute debugging detour happened because the test URL and the client URL were different network interfaces.

The Diagnostic Pattern

When you suspect a service is returning wrong data, work through this checklist before assuming a data or code bug:

Step 1: Identify the exact URL your client is configured to use.

For MCP servers, check ~/.claude/settings.json or the relevant config file:

{
  "mcpServers": {
    "akashic-records": {
      "url": "http://100.91.193.23:8002"
    }
  }
}

Step 2: Hit that exact URL directly.

curl http://100.91.193.23:8002/health
curl http://100.91.193.23:8002/discover

Step 3: Compare against localhost and other interfaces.

curl localhost:8002/health
curl http://192.168.1.x:8002/health

If the responses differ, you have multiple instances. If only one interface responds, your client may be misconfigured or firewalled.

Step 4: Check what is actually bound to that port.

# Show all processes listening on port 8002
lsof -i :8002
# Or more verbosely
ss -tlnp | grep 8002
# On macOS
netstat -an | grep 8002

This reveals the PID and the bound address for every listener. Two entries on the same port means two instances.

Step 5: Verify which instance your client is reaching.

Add a unique identifier to each instance's response — a hostname, PID, or build timestamp. When you query through the client, confirm the identifier matches the instance you expect.

Port Binding Gotchas in Docker

Docker's port publishing syntax silently controls which interface is exposed:

# Publishes to ALL interfaces — accessible from Tailscale, LAN, localhost
ports:
  - "8002:8002"

# Publishes only to loopback — NOT accessible from Tailscale or LAN
ports:
  - "127.0.0.1:8002:8002"

# Publishes only to a specific LAN IP
ports:
  - "192.168.1.100:8002:8002"

If your MCP server or external client needs to reach a Docker container via Tailscale, the container must publish to 0.0.0.0 (the default with no prefix) or specifically to the Tailscale interface IP.

Making Interface Ambiguity Visible

The best long-term fix is to make your service self-identify. A health endpoint that returns the bound interface, PID, and hostname eliminates guessing:

import socket
import os

@app.get("/health")
def health():
    return {
        "status": "ok",
        "hostname": socket.gethostname(),
        "pid": os.getpid(),
        "bind_address": "0.0.0.0",  # or read from config
        "version": os.getenv("GIT_SHA", "unknown"),
        "uptime_seconds": time.time() - START_TIME,
    }

When you curl any interface and see the PID and hostname in the response, you know exactly which instance you are talking to. Two different PIDs on the same port is a smoking gun for a duplicate process problem.

Key Takeaways

The same port number on different network interfaces (localhost vs LAN IP vs Tailscale IP) can be served by completely different processes with different data states.
Testing with localhost when your client is configured to use a Tailscale IP is not a valid smoke test — they are different network paths.
lsof -i :PORT is the fastest way to enumerate all processes bound to a given port and confirm duplicates.
Docker port publishing syntax controls interface binding; the default ("8002:8002") binds to all interfaces, while "127.0.0.1:8002:8002" binds only to loopback.
Health endpoints that return PID and hostname make interface ambiguity immediately visible rather than requiring SSH investigation.

What's Next

Knowing there are two instances on the same port raises the next question: why did the second instance ever build successfully? In Lesson 238, we dig into Docker layer caching — and how a cached COPY step can silently prevent your code fixes and permission changes from making it into the running container.

Two Instances, One Port