Ask Knox

Three Weeks. Clean Logs. Zero Trades.

Watchdog Service checked Agent Framework every five minutes. Log freshness: current. Process alive: yes. HTTP health endpoint: 200. Every five minutes, for three weeks.

In those three weeks, Agent Framework generated 2,646 signals and placed zero trades.

Watchdog Service was doing its job correctly. It was monitoring the right things — for a service that processes requests synchronously and fails with explicit errors. Agent Framework was not that service. Agent Framework was a pipeline where silent architectural failure (score ceiling below threshold) produced valid-looking output at every intermediate stage while producing zero useful downstream artifacts.

This is the health vs. liveness gap. And it is the reason why every trading bot, content pipeline, and agent in Knox's fleet now has a pipeline health check — not just a liveness check.

The Two Check Types

The distinction matters most when debugging "the system seems fine but nothing is happening." Liveness checks all pass — this is expected for a zombie pipeline. The pipeline health check is the one that fails: it will show zero rows in the output table despite weeks of running time.

Building a liveness check: Process alive, log freshness, HTTP health endpoint. These are necessary prerequisites — if they fail, the system is not running. But they are not health checks.

Building a pipeline health check: Query the downstream artifact. SELECT COUNT(*) FROM trades WHERE created_at > NOW() - INTERVAL '1 hour'. If the pipeline runs hourly, this should return non-zero. If it returns zero and the pipeline should have run, you have a zombie.

The time window matters: a pipeline that runs every 4 hours should be checked with a 5-hour window. A pipeline that runs every 5 minutes should be checked with a 10-minute window. The health check fails when no artifact has been produced within 2× the expected cadence.

It helps to name these as a three-rung ladder. L1 (process liveness): is the process running at all? — pgrep, log freshness, a /health 200. L2 (dependency reachability): can the process reach what it needs? — the database responds, the exchange API answers, the queue is connected. L3 (artifact freshness): has the pipeline actually produced fresh downstream output? — a non-zero, recent artifact count. L1 and L2 are necessary prerequisites, but only L3 distinguishes a working pipeline from a zombie. The whole point of this lesson is that most monitoring stops at L1 when it needs to reach L3.

The Zombie Pipeline in Detail

The Agent Framework bot is the canonical zombie pipeline. Working backward from the failed health check (zero trades), every stage looks healthy until you reach the threshold gate. The threshold gate passed — from the perspective of "did it run?" But it blocked every signal from proceeding to order submission, because the architectural ceiling (3.8% maximum achievable score from live components) was permanently below the threshold (40% required score).

The staging analysis reveals the structure: the entire pipeline above the threshold gate is alive and producing. Everything below is empty. The zombie exists at the gate boundary.

The detection query that caught it:

SELECT
  COUNT(*) as total_signals,
  SUM(CASE WHEN status = 'blocked' THEN 1 ELSE 0 END) as blocked,
  SUM(CASE WHEN status = 'ordered' THEN 1 ELSE 0 END) as ordered
FROM signals
WHERE created_at > NOW() - INTERVAL 21 days;

Result: 2,646 total, 2,646 blocked, 0 ordered.

That single query revealed three weeks of zombie operation that every liveness check had missed.

Health Check Patterns by System Type

Trading bots: SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL 24h — expect non-zero in normal market conditions. Alert on zero for >48 hours.

Content pipelines: SELECT COUNT(*) FROM articles WHERE created_at > NOW() - INTERVAL 1d — expect one or more per day if the pipeline runs daily.

AI tutor / embedding pipelines: SELECT MAX(indexed_at) FROM corpus_chunks — the most recent embedding should be within the last ingest cycle.

Agent orchestration: SELECT COUNT(*) FROM completed_tasks WHERE completed_at > NOW() - INTERVAL 1h — tasks should complete at the expected rate.

Memory systems (Semantic memory layer): SELECT COUNT(*) FROM memories WHERE created_at > NOW() - INTERVAL 1d — new memories should be written regularly if agents are active.

The Watchdog Service Upgrade Pattern

Knox's Watchdog Service currently checks log freshness. The upgrade pattern:

# Before: liveness only
def check_health(service: str) -> bool:
    log_path = f"logs/{service}.log"
    mtime = os.path.getmtime(log_path)
    return (time.time() - mtime) < 90  # within 90 seconds

# After: pipeline health included
def check_pipeline_health(service: str) -> bool:
    if service == "perps-bot":
        # Check that orders were generated in last 48h during market hours
        result = db.execute("""
            SELECT COUNT(*) FROM orders
            WHERE created_at > NOW() - INTERVAL '48 hours'
        """).scalar()
        return result > 0
    if service == "blog-autopilot":
        result = db.execute("""
            SELECT COUNT(*) FROM articles
            WHERE created_at > NOW() - INTERVAL '7 days'
        """).scalar()
        return result > 0
    return True  # no artifact check defined — default to liveness only

The pipeline health check runs at a lower frequency than the liveness check — every 2 hours instead of every 5 minutes. It requires database access. But it is the only check that would have caught the Agent Framework zombie after day one, not day twenty-one.

Now perform the upgrade yourself: take a /health that returns a reassuring 200 for a zombie and replace it with an L3 check that proves the artifact grew and is fresh.

What a Pipeline Health Check Cannot Catch

Pipeline health checks prove that work is flowing. They do not prove that work is correct. A trading bot placing 100 losing trades per day passes the pipeline health check. The distinction matters for higher-level monitoring:

Pipeline health check: Is work flowing? (artifact count > 0) Quality check: Is the work correct? (profit rate, error rate, human review)

Both layers are required. The pipeline health check prevents zombie pipelines. The quality check prevents zombie-work — systems that produce output but produce wrong output. For trading systems, quality monitoring (P&L trend) runs in parallel with pipeline health checks (order count).

Watchdog Service covers liveness. The pipeline health check covers flow. Quality monitoring is the third layer — and the subject of the monitoring and observability track.