Capstone — Deploy an Observable LLM Service

From "It Runs on My Machine" to a Production Service

This track opened with a single uncomfortable claim: "it runs on my machine" is not production. Eleven lessons later, you have every piece needed to retire that excuse for good. You containerized an AI service (482), orchestrated it on Kubernetes (483), scheduled it onto GPUs and made it autoscale (484), served a real model with vLLM (485), wired RAG and caching (486), reasoned about its economics (487), described it as code (488), instrumented the three pillars (489), defined SLOs against artifact health (490), and learned to ship it without breaking it (491).

The capstone assembles all of it into one graded build. This is the track's completion project: a single, observable LLM service that you stand up, instrument, and defend with a real SLO — end to end, on a running system.

The System You Are Building

The architecture above is not a reference poster — it is the literal box diagram of what you ship. Trace the request path: a client (an academy page, an agent) calls the gateway, which authenticates, rate-limits, routes, exposes /metrics, and starts an OpenTelemetry trace. The gateway checks the cache — a hit skips the GPU entirely — and on a miss routes to the vLLM pods running on GPU nodes with continuous batching and HPA autoscaling. The pods pull context from the vector DB (the RAG retrieval path, Semantic Memory Layer-style: embed → top-k → inject). And wrapping every one of those boxes is the observability plane: Prometheus pulling /metrics, Grafana charting it, Jaeger collecting OTel traces, and a burn-rate alert watching the SLO.

The single most important structural fact: the observability plane wraps the whole request path — it is not a box at the end. Gateway, pods, vector DB, and cache all expose /metrics and emit spans. An un-instrumented hop is a blind spot you will discover at the worst possible moment, when latency triples and the trace shows time disappearing into a service you forgot to instrument.

The Graded Checklist

Six groups, twenty points each, one hundred and twenty total. Every box is something you show, on a running system.

The three groups where operators consistently stop early are OBSERVED, SLO-DEFINED, and DRIFT-DETECTED — and not by accident. They are the least visible to a casual demo (the service "works" without them) and the most load-bearing over time (the service silently degrades without them). The Semantic Memory Layer silent-failure incident and Mission Control's artifact-health checks both exist precisely because someone, somewhere, declared a service "done" at the SERVED stage and shipped a blind spot. Do not be that someone on your capstone.

The Drift-Detection Rubric Group

The sixth group — DRIFT-DETECTED — exists because a service that passes OBSERVED and SLO-DEFINED can still fail silently in a category those groups cannot catch: the model itself changes.

Provider silent upgrades are real. A cloud-hosted model endpoint you call today may silently route to a different model version tomorrow — same API, different behavior, different output distributions. A latency or cost regression that stays under your burn-rate threshold is invisible until you look at the trend. And an output-distribution shift (responses getting shorter, refusal rates increasing, quality scores drifting) can erode a product without any alert firing, because no SLO was defined against model behavior.

Drift detection requires three things to earn full credit:

1. A golden baseline. At a known-good model version, capture a sample of output distributions (token counts, quality scores, refusal rates) or a cost/latency profile. This baseline is the comparison anchor for every future deployment. Without it, there is nothing to compare drift against.

2. A first-class drift signal. The running system must compute and expose — as a Prometheus metric or equivalent — at least one of: output-distribution shift vs the baseline (e.g. KL divergence or percentile shift on a quality scorer), a model version fingerprint check (detect when the provider silently changes the model behind the endpoint), or a latency/cost trend alert that fires before the SLO burn rate is reached.

3. Demonstrated in a test. Like the burn-rate alert in SLO-DEFINED, the drift alert must actually fire during your capstone submission window. Inject a model swap (swap the serving endpoint to a different model or a mock that returns a shifted distribution) and show the alert triggering.

The DRIFT-DETECTED group is the production operator's acknowledgment that reliability is not only about your infrastructure — it is also about the model behavior your infrastructure is serving. A service that cannot detect that its model changed is not observable. It is observed from one angle.

This is the graded build, decomposed into a single scaffold with TODOs spanning the full stack — Dockerfile to Kubernetes manifest to serving config to SLO. Fill every TODO, stand the system up, and demonstrate each rubric group on the running system.

The Deliverable

Submit a GitHub repo containing: the Dockerfile, the Kubernetes Deployment/Service/HPA manifests, the serving + RAG/cache config, the Prometheus scrape config and a Grafana dashboard JSON, the SLO spec, the drift-detection baseline capture and alert config, and a short CAPSTONE.md. The write-up must include evidence for each rubric group — docker build output, kubectl get pods showing pods on GPU nodes, a Grafana screenshot with rate/p99/GPU-util/cache-hit panels, a Jaeger trace spanning the full request path, a screenshot of the burn-rate alert firing during your injected-fault test window, and a screenshot of the drift alert firing when you inject a model swap or distribution shift. Describe which group revealed the most interesting failure during your build.

What You Have Become

You started this track as an operator who could make an AI service run. You finish it as one who can make it production: containerized with a non-root image, orchestrated onto GPUs with autoscaling, served efficiently with vLLM and caching, observed across all three pillars, defended with an SLO measured against artifact health, shipped with a strategy chosen by what rollback costs, and instrumented to detect when the model itself changes. That last discipline — drift detection — is what separates a service that was reliable at launch from one that stays reliable as providers evolve. That is the operator's layer: the discipline that turns a model on a laptop into a service the rest of the empire can depend on month after month. The same discipline runs Semantic Memory Layer, Mission Control, and the trading fleet today; now it runs whatever you build next.