Observability for AI Systems

Standard APM Is Not Enough

Datadog, New Relic, and every other APM tool in your stack answers three questions: is the service responding, how fast, and with what error rate? These are necessary. They are not sufficient for AI systems.

Standard APM cannot tell you whether the AI said something useful, how much that specific call cost, which classifier branch it took, or whether output quality has degraded since last Tuesday. These are the questions that actually determine whether your AI product is healthy.

The four pillars of AI observability are: latency (is it fast enough?), cost (is it cheap enough?), quality (is it good enough?), and safety (did anything problematic get said?). Standard APM covers latency. The other three require custom instrumentation.

The Minimum Viable Log Record

Every AI call in production should produce one structured log record. This is not an analytics platform — it is a console.log with structured fields that your log aggregator can query.

The minimum viable fields:

console.log(JSON.stringify({
  timestamp: new Date().toISOString(),
  user_id: userId,
  session_id: sessionId,
  trace_id: traceId,           // links multi-step calls
  model: 'claude-sonnet-4-6',
  input_hash: hash(inputText), // NOT the raw text — hash for privacy
  output_hash: hash(outputText),
  tokens_in: usage.input_tokens,
  tokens_out: usage.output_tokens,
  cost_estimate: (usage.input_tokens / 1000 * 0.003) + (usage.output_tokens / 1000 * 0.015),
  latency_ms: Date.now() - startTime,
  cache_hit: cacheHit,
  tool_calls_made: toolCallNames,
  classifier_mode: classifierOutput, // if using a classifier
}))

Notice what is not here: the raw input text and the raw output text. Storing full message content in your log aggregator creates privacy liability and drives up storage costs. The input_hash and output_hash let you correlate logs to specific requests for debugging without storing the content itself.

The cost_estimate field is the one most developers omit. Without it, you cannot answer: which feature costs the most? Which user is most expensive? Did your caching optimization actually reduce costs? Provider billing APIs give you account totals — not per-feature or per-user breakdowns.

Tracing Multi-Step Flows

When one request triggers multiple AI calls — a classifier that routes to a specialist that calls a tool — they need a shared trace_id. This lets you reconstruct the full call graph when debugging.

The pattern: generate a UUID at the top of the request handler and pass it through every subsequent call. Each call logs the same trace_id. When a user reports a bad response, you query logs by trace_id and see every AI call that contributed to it — classifier output, specialist call, tool calls, latency at each step.

Without trace_id, debugging a multi-step flow means correlating timestamps and session IDs and hoping nothing overlapped. With it, the full picture is one query.

The Cost Dashboard You Actually Need

Five panels that tell you whether your AI product is financially healthy:

Per-user spend (daily). Who are your most expensive users? Power users on the free tier who have found a way to maximize token consumption — they often appear in the top 5. This is also where you catch abuse: a single user spending $2/day when average is $0.05 is a signal worth investigating.

Per-feature spend. Which feature costs the most to serve? The answer is often surprising. Code generation endpoints cost 5-10x more than support endpoints because output tokens are more expensive than input tokens and code generators use more of them. This data drives pricing and feature-gate decisions.

Daily burn rate with trend. Is cost per day increasing, flat, or decreasing? Flat is good. Increasing faster than user growth means cost per user is rising — a margin problem. Decreasing means your caching and optimization work is paying off.

Cache hit rate with trend. Should be increasing over time as your cache warms up. A sudden drop is a signal: something changed in request patterns, your cache TTL is too short, or the cache itself has a bug.

Average tokens per request with trend. Should be stable or decreasing. If it is increasing, something changed: conversation histories are growing, retrieved context is getting larger, or users are asking longer questions. Each of these has a different fix.

Quality Monitoring Without Human Reviewers

Hiring humans to review AI responses at scale is expensive and slow. You can approximate quality signals from the data you already have.

Output length distribution. Very short responses (< 50 tokens) often indicate a refusal, an error message, or a confused model. Very long responses (> 2,000 tokens for a support tool) often indicate the model going off the rails or a max_tokens misconfiguration. Plot the distribution over time — changes in the median or tails are quality signals.

Error rate by classifier mode. If you use a classifier to route requests, break down error rates per mode. A 12% error rate on the 'medical' specialist versus 0.8% on 'general' tells you exactly where to investigate.

Session completion rate. Users who complete their session task versus users who abandon mid-session. A drop in completion rate is a leading indicator of quality problems — users are bailing because they are not getting useful responses. This signal arrives before complaint tickets.

The Alert That Pays for Itself

Three alerts cover 90% of AI-specific incidents:

Per-user daily cost spike at 10x average. If your average user costs $0.05/day, alert when any user exceeds $0.50/day. This catches: runaway retry loops (one bug that calls the API 100 times instead of once), abuse (a bot using your free tier to run inference), and bugs in conversation history management (histories growing unbounded).

Error rate above 5%. A 5% error rate on AI calls means 1 in 20 users is getting a failure response. This threshold triggers before users notice at scale.

Cache hit rate drop above 20% from baseline. A 20% drop means something changed in your traffic patterns or your cache layer. It could be a cache bug, a new feature that bypasses the cache, or a change in user behavior.

Knox's Mission Control tracks per-service API costs daily across 40+ connected services. When the trading bot observability service spiked 5x in one day — a misconfigured logging loop was calling an LLM on every price tick — the alert fired within 2 hours. Without it, the month-end bill would have been $400 higher and the root cause would have taken days to find.

Implementation Path

Start with structured logging on every AI call — the minimum viable log record above. Add the cost dashboard panels using whatever log aggregation your hosting platform provides (Vercel logs, Railway, or a self-hosted solution). Set the three core alerts. Add tracing for multi-step flows as you build them.

The full quality monitoring suite — output classifiers, A/B test harnesses, semantic drift detection — is Phase 2 work. It pays off once you have enough traffic to make the signal meaningful. The foundation above is what you need before you have that traffic.

For production observability patterns from Knox's Tesseract Intelligence and Mission Control deployments, jeremyknox.ai covers the architecture in depth. The InDecision analytics engine uses this exact observability stack across all its AI recommendation endpoints.