Observability — The Three Pillars

You Cannot Operate What You Cannot See

In the “Serving an LLM with vLLM” lesson you served a model with vLLM. In the previous lesson you described the whole stack as code. The service runs. Requests flow. And then, at 3am, the p99 latency triples and you have no idea why — because "it runs" and "you can see inside it when it breaks" are two completely different properties.

Observability is the second one. It is the ability to ask arbitrary questions about your system's internal state from the outside, without shipping new code to answer them. For an LLM service — where the failure modes are slow token generation, GPU memory pressure, cache misses, and stale retrieval — the questions you will need to ask are not knowable in advance. So you instrument broadly and answer them later.

The discipline rests on three pillars. Each answers a different question. None substitutes for another.

Metrics, Logs, Traces — The Division of Labor

Metrics are numeric time series. request_rate, p99_latency_ms, gpu_util_pct, cache_hit_ratio. They are cheap because they are fixed-cardinality: a gauge is one number per scrape, downsampled and kept for months. Metrics answer "is something wrong, and how much?" They power dashboards, SLOs, and burn-rate alerts. This is where you look first because this is where the alert came from.

Logs are structured events — one timestamped line per thing that happened. tool_invoked, write_failed, retry_attempt, auth_denied. They are expensive at volume and high-cardinality, so you keep them for days, not months. Logs answer "what exactly happened, in order?" You go here after a metric alert fires, to find the specific event that explains the spike.

Traces follow one request across every service it touches. A single trace for an inference request might span gateway → vLLM forward pass → vector DB retrieval → cache check, each as a parent/child span with its own duration. Traces are sampled — keep 1-10%, keep 100% on errors — because storing every span is ruinously expensive. Traces answer "where in the request did the time go?" When p99 triples, the trace tells you whether it was the model or the retrieval, in seconds.

This maps directly onto a real production stack. Mission Control is the metrics-and-dashboard layer for your production ecosystem — it is where a gauge like chunks_indexed or a trading bot's last-trade timestamp becomes a number you can chart and alert on. The structured logs of each service (for example, logs/service.log, logs/watchdog.log at your deploy target) are the second pillar — the per-event record you grep after an alert. The pattern you are learning here is the same one that already runs a production fleet; you are just making it explicit and portable.

How Prometheus Actually Collects Metrics

The single most important thing to understand about Prometheus is the direction of the arrow: Prometheus pulls. Your service never pushes.

Your service exposes an HTTP endpoint — /metrics — that renders its current counters and gauges as plain text. Prometheus is configured with a list of targets (resolved via Kubernetes service discovery in production) and a scrape interval, typically 15 seconds. On every tick it polls every target's /metrics, appends the samples to its local time-series database (the TSDB), and moves on. Grafana then runs PromQL queries against that TSDB for dashboards, and Alertmanager evaluates alert rules against the same data.

The pull model is not an arbitrary design choice — it buys you target liveness for free. Every scrape either succeeds or fails, and Prometheus records that as the synthetic up metric. If a vLLM pod dies, its next scrape fails, up goes to 0, and you can alert on that with zero extra instrumentation. A push-based system cannot distinguish "healthy but quiet" from "dead and silent" without building that distinction yourself.

The one place pull breaks down is short-lived jobs — a cron that finishes in two seconds will be dead before any scrape lands. Those push to a Pushgateway, which holds the last value so Prometheus can scrape it. That is the exception that proves the rule: even the push path terminates in a pull.

Now wire it: a Prometheus scrape config that pulls /metrics from the inference service every 15s, plus the PromQL for the panel that matters — p99 latency, computed from the histogram, not read off a gauge.

What to Instrument on an LLM Service

For an inference service, instrument these from day one, because they are the things that break:

• http_requests_total (counter, labelled by route and status) — request rate and error rate fall out of this.
• inference_latency_seconds (histogram) — gives you p50/p95/p99 for the SLOs you will define in the next lesson. Use a histogram, not a gauge, so you can compute quantiles server-side.
• gpu_utilization_ratio and gpu_memory_used_bytes (gauges) — the GPU is your most expensive resource and your most common bottleneck.
• cache_hit_ratio (gauge) — a falling cache hit rate silently shifts load onto the GPU and is invisible without this metric.
• tokens_generated_total (counter) — the unit your cost is denominated in.

Notice what is not here: prompt text, completion text, user IDs. Those are high-cardinality and belong in logs (and even there, redacted). The single fastest way to kill a Prometheus instance is to put a unique string — a prompt, a request ID, an embedding — into a metric label. Cardinality is the enemy; keep labels to a small, bounded set.

This connects back to a hard lesson from the Semantic Memory Layer build: a health endpoint that returns 200 OK proves only that the process is alive. It does not prove that chunks_indexed is growing or that the last ingest ran. That distinction — liveness versus genuine pipeline health — is the entire subject of the next lesson, and it is why your metrics have to read the artifact, not just the heartbeat. You will also find the same observability discipline applied across the ecosystem at tesseractintelligence.io, where signal freshness is the product.

The Operator's Takeaway

Observability is not a dashboard you bolt on at the end. It is three instrumented surfaces — metrics, logs, traces — wired in before you need them, because the questions you will need to answer at 3am are not knowable at deploy time. Metrics catch the problem and quantify it. Logs explain it. Traces locate it. Wire all three, expose /metrics for Prometheus to pull, and keep your metric labels low-cardinality. Do that, and the next outage is a ten-minute investigation instead of a two-hour guessing game.