SLOs, Error Budgets & On-Call

"Pretty Reliable" Is Not an Engineering Target

Ask an operator how reliable their service is and you usually get an adjective: "pretty solid," "mostly up," "we had a rough week." Adjectives do not survive an incident review and they do not tell you whether you can ship on Friday. Reliability has to become a number — one you measure, commit to, and spend.

That is what the SLO stack does. It converts a vague intention into a chain of concrete artifacts: an indicator you measure, an objective you commit to, a budget you are allowed to burn, and an alert that fires when you are burning it too fast.

SLI → SLO → Error Budget → Burn Rate

The SLI (Service Level Indicator) is a measured ratio of good events to total events. The art is picking one the user actually feels. For an inference service: good = requests with latency < 2s AND status 2xx, and SLI = good_requests / total_requests. Not CPU. Not pod count. The thing the user experiences.

The SLO (Service Level Objective) is the target you commit to over a rolling window: "99.5% of inference requests complete in under 2 seconds, measured over a 30-day rolling window." The window matters as much as the percentage — a 99.5% daily SLO and a 99.5% monthly SLO permit very different incident sizes.

The error budget is the arithmetic complement: 100% − SLO. A 99.5% SLO yields a 0.5% error budget, which over 30 days is roughly 3 hours and 36 minutes of permitted badness. This is the number that changes how teams behave. Budget remaining means you ship freely — deploys, canaries, experiments all spend from it. Budget exhausted means you freeze features and spend your effort on reliability until the budget recovers. The number decides, not the loudest voice in the room.

Burn-rate alerting is the final piece. You do not page on every error — that is how you train your on-call to ignore the pager. You page on the rate at which the budget is draining. A 1× burn would exactly exhaust the budget over the full window. A 14.4× burn would exhaust a 30-day budget in about two days — that is a fast burn and it pages. A sustained 1×–3× burn is a slow leak and it files a ticket. This multi-window approach (a fast-burn page plus a slow-burn ticket) is the standard Google SRE pattern, and it is what keeps the pager meaningful.

The Trap: Measuring the Heartbeat Instead of the Artifact

Here is where most SLOs quietly lie. You define a clean availability SLO, you wire the SLI to your health endpoint, you watch a green dashboard — and you sleep through a 30-hour outage. Because the health endpoint you measured was a liveness probe, and liveness is not health.

There are three levels of health check, and they prove progressively more:

• L1 — Liveness: GET /healthz → 200. Proves the process is running. That is all it proves.
• L2 — Readiness: dependencies reachable — db_connected, gpu_visible. Proves it could accept a request.
• L3 — Artifact health: is the downstream artifact fresh and growing? chunks_indexed increased, last_run is within the expected window. Proves work is actually being done.

The dangerous quadrant is L1 green + L3 stale. The process answers 200, the dependencies are connected, every dashboard is green — and the product is broken because the artifact stopped updating. This is the exact shape of the Semantic Memory Layer incident: chunks_indexed flatlined when the ingest cron died, while the server returned 200 OK the entire time. A liveness-based SLI would have reported 99.99% availability straight through it.

The rule that falls out: your availability SLI must read L3, not L1. Measure last_indexed_at within 24h and chunks_indexed growing, not just "process answers." This is the same discipline Mission Control applies to the whole ecosystem — a service is "healthy" only when its artifact proves it, and that observability-SLO pairing is what the Mission Control dashboards are built to enforce. The longer argument on why heartbeats deceive is the one this lesson makes: a heartbeat answers "is the process up?" while a real availability SLI answers "is the artifact fresh?" — and only the second one catches a silent indexing failure.

Build It: Define a Real SLO and a Burn-Rate Alert

Theory becomes muscle memory when you write the actual rule. Define an SLO for an inference service and the fast-burn alert that protects it — and make the SLI read the artifact, not the heartbeat.

The Operator's Takeaway

An SLO turns reliability from an adjective into a number, and the error budget turns that number into something you can spend: ship while the budget holds, freeze and harden when it runs out. Burn-rate alerting keeps the pager honest by firing on the speed of spend, not on every error — page on a fast burn, ticket on a slow one. And the whole apparatus is worthless if you measure the heartbeat instead of the artifact: an SLI wired to a liveness probe will sleep through the exact silent outage your users feel most. Define the SLI against L3. Measure what the user actually experiences. Everything else follows.